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1. Tntroductlon (J» Nievergelt) 

From 1972 to 76 the Department of Computer Science has 
been heavily involved In a project to develop an automated instructional 
system for teaching coniputer programming. After four years of 
implementation, with an effort in excess of 25 man-years vhich produced 
approximately a million words of code, our system ACSES (Automated 
Conputer Science Education System) is now in routine large-scale 
use^ assuming about 50f) of the tecuihing load in various introductory 
computer science courses, with a total enrollment ctf over 1^00 stxidents 
per semester. 

ACSES runs on the FIATO IV system developed by the Computer- 
based Educ&tlon Research Laboratory at the Uhlversity of Illinois. 
The approximately 1000 terminals across the country attached to the 
Illinois FIATO s;rstem have permitted a smaller scale use of ACSES in 
some other schools. It is to be expected that the use of ACSES will 
continue to increase, in our own courses as well as elsewhere. Vlhile 
the initial development of ACSES is now complete, expanded use requires 
a continuing effort to maintain the system: adapting it to changes and 
new features of the FIATO system, adding new instructional material, and; 
most importeuit, improving -^cisting material on the basis of experience 
in actual instruct ioncQ. use. 

The purpose of this report is to document the ACSES project: 
to serve as a case study in the design, implementation, and use of a 
major effort in computer-aided instruction. This introduction is a 
concise description of the ACSES projects it presents the motivation 
for starting the project^ the design criteria, the components of the 
resulting system, the experience gained during the implementation and 



use of ACSES. The remainder of the report describes various aspects of 
the project in more detail. 

Why ACSES? 

Computers are playing an increasingly pervasive role in our 
society, and this fact leads directly to a rapidly Increasing demand 
for basic computer science education. This demand arises from two 
sources. 

First, the demajid for computer profess iorietls continues to 
grow. The most widely accepted projections show a doubling of the total 
demand for systems ansQysts, programmers, computer operators and 
associated technicians during the next five years. 

Secondly, it seems reasonable to expect that most people will 
be required to interact with computers in their daily work within a 
decade or two. Even people not directly concerned with computers should 
have some understanding of them because an enlightened public will be 
essential if we are to make intelligent decisions concerning the future 
role of computers in our society. Currently many citizens view computers 
with indifference, while others fear them as the xiltlmate threat to their 
privacy, security, and dignity. Such attitudes will clearly not suffice. 
Hence, it is important that every educated person have some xinderstcuiding 
of the principles underlying computers and their implications for society, 
as well as some skill in their use. At the very least, everyone shoxild 
have the opportunity to acquire this knowledge in a convenient way. 

Recognizing the inportance of ''computer literacy," the 
President's Science Advisory Committee recommended in their I967 report 
that 75?6 of all college students should have a meaningful exposure to 
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computers. Even though this percentage has not yet been attained, the 
demand for instruction in basic computer science at our universities, 
colleges, junior colleges, and private electronic data processing 
schools is enormous. These institutions have relied on the traditional 
instructional approach of lecture-discussion-laboratory. This approach 
suffers from several defects, particularly when ?.t involves large numbers 
of stiidents. It is not particularly suited to the subject and is, 
therefore, the cause of student dissatisfact5.on. Learning to program 
requires active participation and intense effort on the part of the 
student— two things that are not encouraged by the lecture type of 
instruction. An individual tutor for eveiy student would.be ideal for 
learning a skill such as programming, but such a mode of instruction is 
obviously economically not feasible when one aims at a mass-education 
program. 

In addition, the traditional lecture-based approach can: only 
reach a limited audience. In particular, it excludes all people whose 
professional duties prevent them from attending school for any length 
of time. It is to be expected that there will be a large demand for 
basic computer science courses in connection with continuing adult ^ 
education programs. The situation ndiere somebody suddenly fiUids that 
he should taiow something about computers in order to remain effective 
on his job, will become an increasingly familiar event. 

All of these long-range considerations, in addition to out* 
own experience in teaching introductory ccxnputer science at the 
University of Illinois to about 2000 students of widely different 
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backgrounds every semester, led to the initiation of a laxGe-sc€d.e 
project to automate introductory computer science courses, the results 

^ 

of vhich are described in this report. 

The PIATO IV system being developed by the Corap:iter-based 
Education Research Laboratory at the University of Illinois gave us 
a unique opportunity to develop an automated course consisting of 
CAI- lessons about computers and of supporting software, and to try this 
system out on a large audience of diverse educational background. The 
PIATO IV system, while centered at the University of Illinois, serves 
nearly 1000 terminals located at schools and colleges with very different 
typ6i^' of student populations, and with wide geograjtoic dispersion. 

Potentially, computer-assisted instruction (CAl) has many 
advantages: It can provide truly individualized instruction by allowing 
students to study sequences of lessons tailored to their needs and at 
their own pace; given a suitable terminal network, it can reach a wide 
audience and, in the forseeable future, do so at low cost. Becatise of 
its potential cost effectiveness, CAI may become the cheapest wajy for 
schools and colleges who do not as yet o'ffer computer science courses 
to institute programs in computer studies. However, these potent iial 
advantages of CAI will be realized only if much more research and a large 
scale development effort is carried out. We view our project as a 
contribution towurds the goal of demonstrating the feasibility of a 
CAI-based approach to the problem of mass- education in basic computer 
science. 
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Design crit^^riEj and the resxg-t^istg structure of ACSES 

ACSES, our automated computer science education system 
developed on PLATO IV, is" designed to be usable in tvo' modes, according 
to the two purposes it is intended to serve: supplementary instruction 
^,,^*HOur owi university, and main-line instruction at remote sites. 

a) The partially automated mode for supplementary instruction 

In our Cc»nputer Science Department an instructor is still 
responsible for a course, and CAI lessons are used in an adjunct mode. 
He may discuss a problem in classroom, and then refer the students to an 
appropriate lesson that gives more detail, exannaes, or allows the f^tulent 
to practice or solve problems on the computer. In the case of our con^niter 
science lessons, practicing and problem solving usually means that the 
student must write and execute a small program. He is able to do so at 
the same terminal, and switch easily from lesson taking to programming, 
and back. 

Thus, in order to operate a partially automated introductory 
computer science course, one needs primarily: 

— a library of lessons, covering several programming languages, computing 
techniques, and application areas 

— a completely self-contained interactive programming system for the 
preparation, execution and debugging of programs written by students 
in any of the languages covered by the lessons. 

^an^exam system, to automatically generate problems according to 
an instructor's specification, to grade the student's solution, 
and administer the exam (data collectlon and security aspects). 

b) the fully automated mode f or main^line instruction 

We expect that the demand for basic computer science courses 
on the part of high schools, junior colleges, and continuing adult 



education vHJL grow rapidly in the near future. In these settings, 
there may not be an instructor available who can guide the student *s 
course of study and fill Jiny gaps that might be present in the lesson 
jnaterial. Hence in this setting the system has to be usable ^ a 
fully autcmated mode, and ACSES makes this possible primarily 
uy providing: 

— a conversational adylce-giving and information retrieval system 

to guide the student throtigh the library of lessons, based on his 

goals and past performance 
a communication system that allows a student to contact a human 

tutor for help and advice, from any terminal connected to. the 

PIATO system. — " 

We consider our project to be a major research effort to 
investigate the extent to which a large introductory course can be ; 
automated. The objective has not been to take one of our existing 
-CS courses and put it on PIATO as it is. Rather, we want .to t\im 
PLATO terminal into a rich environmeifit, analogous to a convehtiohal 
library and laboratory, where you have at . your ftogert 

things for learning about computer science, and for practicing immediately 
^what you have learned. It is the student and his instructor who decide 
which one of these things they want to use. 

Also, the project has an additional goal; namely tc serve as 
a stimulating environment for computer science reisearch in a variety 
of areas: compilers. Information systems, artificial intielligerice. 
This last point may deserve some explanation, since the miscoriception 
is widespread that lesson writing is a routine activity. It can be, if 
one creates poor lessons. It can also be a task as challenging as you 



wiish to Mike it, if you view a lesson as an interactive program that 
has a certain dc»nain of knowledge, and is able to coimminlcate with 
students about the knowledge it has. We fe^l'^that if this necessary 
ingenuity and effort are put into the design of ah instructional 
system and its supporfcing software, such a system can be made to 
provide an educationsLt experience superior to the one a student has 
in a large intxx>ductory course taught in* the conventional lectxire- 
based manner. 

Experience gained during, the i':trpiteaentation 

From the technical point of view> the design and Implementation 
of a computer-based instructional system is no different than the 
development of a large system for some othei^ applicatlonv^^ 
software components dominate: conqpilers, interpreters, ^filing^ 
data management. A high level programming language is desiiral:K^ almost 
a necessity. Tutor, the only programmiiig lar^^ 

of instructional material on PIATO> is a very high .1^* *4 i^aiguage f pr 
programming man-machine dialogs, but is rudimentaxy with^re^ . 
structuring large software systems into self-contained modules, and 
designing clean interfaces between them. It is well suited to programming 
small or medium-sised (a few thousand words of source or ob^Ject code) 
conventional lessons, which are I/O. intensive and> as prdgrams>> have a 
simia.e structure. It is considerably less we^ . suited :f 
large, complex- prpgramis, such as the; compiler system, the/ 1^ 
and advising system, and the exam system of ACSES. , .:'y-.-. ^ 

~ Another hindrance whith affects^^the performance of our^ cc^^ 
programs, in particular, the interactive compilers, is'that PIATO limits 



one user's program to 2 msec of CPU time per clock second under normal 
aystem loads. This is sufficient for the conventional lessons > but .it 
causes the response time in more complex programs to be irritatlngly 

slow, /• --^^ ■ 

With respect to educational and psjrchological aspects of 
a computer-based instixictibnaJL system> it nw said that there is no 
systematic boctjr of knowledge to guide the designer of such a syBt^^ 
The voliuainous literefcure on CAI, to the extent that it reports : 
experiments to determine what are effective eriyijc^ 
treats detailed aspects in isolaticr), ijuader controlled cohdit^ 
do not apply to the varied situfirtiions that occur when^ M 
is given by computer. The best advice is to try everything i^^ 
instruction as soon as possible, to be prepared to naa^ : 
modifications in response to feedbax* frcm the users,; 
unsuccessful material. The most productive point of view seenis^^^t^ 
to consider a computer-driven graphics tenatoal as' a medium w^ " 
properties,, and to develop a crafts ot writing interactive pTOprcuns for 
coimminication of all kinds, educational or other. 

Experience using ACSES 

Repeated experiments, questionnaires, and plain observation 
makes it clear that a large majority of the students like the experience 
of working on PIATO, and that many prefer studying a lesson to attending - 
a lecture in the large classes typical of introductory ccniops 
is the most conclusive finding: of our effort to evaluate ACSES in instruction. 
In contrast to' this, we have no conclusive data for compearing the 
performance of students of f and on PLATO - usually no difTerence between 
the two groups vere detected,. A common- sense conclusion is that others - 



f^iotersj aueh as the instructor, and tbe notiT«;tlon^^.s^ 
a nieVst^ influence on the student * s perf oxMace jthaa the 
d^ 

Conclusion ' 

project can serve a«; a benelnurlr to Indicate vhat 
effort is required to develop, nalntaln, and luie a conpater-based 
Instructional system caprina^ of assuming a large part of the teachitig 
load in large introductory courses. We hope that others aiy hwiefit 
from the experiences described in this report.' 
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2. Thft library of lessons (H. 0. Friedman, Jr.) 

The largest eonponent of the ACSE8 system is our library of 
Instructional lessons. We have over 135 Individual lessons, groupisd 
Into about 20 areas (Appendix 1). The majority of these lessons have 
-been created by students as term projects In courses on conputar^asslsied 
Instruction, or as other academic activities, such as theses. As a reiult 
of this origin, our lessons exhibit a vide variance In level of ^lallty. 

A typical lesson in our llbraxy Is a program nhidh presents 
some information in graphical and textual form, and provides sotae opportiinity 
for the student to practice and assess his understanding of the stibject 
matter by solving problems and answering questions. It is cooparable in 
scope to a tutorial article or short chapter in a book, and may occupy a 
student for half an hour to a few hours. 

According to this analogy, our library of 135 lessons represents 
a small publishing enterprise. The effort required in creating and 
maintaining a collection of Interactive programs for instruction (course- 
ware), however, is significantly larger than "it is for a conrparable amount 
of material in textual form. The main reason Why courseware development 
is so labor-intensive is that, on a medium as powerful as a computer with 
a graphics terminal, one wants to use techniques of presentation and inter- 
action that are much more elaborate than anything that could be done on 
paper, iteplementatlon of these techniques takes a lot of progranmlng time. 
Moreover, lessons using elaborate techniques must often be revised several 
times, because little experience is avaiUble for guiding one on how to 
use these techniques effectively at first attempt. 

This need for repeated revision based on experience with actual 
ins,truotion, and the significant amount of effort re<mired for each revision, 
are the causes for the fact that only V fraction of our lessons are anywhere 

-10- 
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close to their "flnftl form"* As mi^t be expected^ those lessons vfaich 
have been used most In orir own Instruction^ particuliuly t 
sequence, are the most polished* 

— Among the improvements Jto the lessons has been a certain smount 
of standardization. For eatample, the function of the keys" that allow a 
student to, control his path throu£^ a lesson hias^l^^ 
all CS lessons, so that a student l^d has god^^^^^ bas 
become familiar with most of the control options in all lessons and can 
proceed throufifh subsequent lessons more easily. These convmxtiohp are 
described in lesson csauthors; standard pieces of code, character sets, 
and micro tables have been colledted in lesson cslibraxyj other coding 
suggestions have been collected in lessOn cscode* 

Also, several communication lessons have been complebed# Note- 
file csnotes serves a *1)ulletin board" function between authors. Lesson 
cscomments serves to record r.emarks made by students taking instructional 
lessons^ for feedback to the authors of the lessons. Lesson cstalk allows 
real-time communication between an instructor and each of several students 
this allows human assistance to a student by an instructor who might 
be located at a different PLATO site. 

Finally, H. 0. Friedman, who manages the library of lessons, 
has written a router lesson, csrouter, which allows students access to the 
library and maintains records on Vhich lessons the student has entered, 
which lessons have been completed (as set by eaoh individual lesson), 
elapsed time spent in each lesson^ etc. 
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P^3. / <ia^ syiteia (GAPS ) (T.^ R^ ; Wii^^ P^yii, 

^ design goad of CAK 

slTOdeiiti pTOg3?fittflm3^ errors imd heljp- st^<tent,;U^ " 

*?recover " from detected errors r li^st^ it interacts; "irt^ 
- to Inform him !of Ms error and attenrpts^ t^^ ^^^-^^^^ • ^^^^^^ 

kbout of Correcting the-^inrqr. v ^Ch 
analyze the situation and. then repair his program.^ 
is provided both for compile-time errors and run-time errors. 

Most components of CAPS are table driven. This, pertait^ 
a compact implementation and provides a convenient mechanism for 
supporting the different languages that are tau^t at the Uhlversity. 
Tables for FORTRAN, PL/l and COBOL have been written and pu^ 
classroom use. Tables for LISP, PASCAL, BASIC and pOBOL^^ 
preparation. Severe time and space , constraints imposed by PLATO' 
(as it services 500 terminals concurrently) limit the capabilities of 
CAPS to simple programs using only subsets of the abov^ profipramnlng 
languages. CAPS is sufficient to handle the computing requirM^^ 
for the greater part of a first semester progranining course. ; Sul^set 
number five of 8P/k [5], for example, is typical of the language featxires 
supported by CAPS. 

3.2. System organization 

The principle modules of the^system are a program ^editor, a 
syntactic and static semantic error diagnostician, an interpreter for 
each language supported, a run-time erijor analyzer, a user program file 
manager, and a system's table builder and file manager (Figure l). 

-12- , , •/ 
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Each of .these modtiles is written in TUTOH - a FORTRAN level programming 
Tfthgiiage and the only one used on FIATO IV [lo] and holds the 
position of a lesson within the PLATO IV system. All lessons are 
poteniially. reentrant pure procedures and they are_the unit of multi- 
programidjig in the PLATO 17 system. 

/ Control ot the system is distributed throughout the modules, 
but the student is never awiare of the system modularity etnd never has 
to remember connand syntax, because each time the system is; ready for 
a command, the module that will ihterpz^et the commwd (Us^^ 
of possible actions. Usually pressing one key will initiate a new 
action. 

Daring a programming session student-specific commanciatlon 
between modules is througih the block of storage (1500 words) assigned 
to each terminal. This block contains the only internal representation 
of the student *s progrcun, the associated symbol table entries and 
miscellaneous information identifying the student and the lianguage he 
is using. The internal representation of the student * s prcgr em is us^d 
both to regenerate the listing on the screen and as the "object code" : 
for the language interpreter. It is a simple tokenization of the inpui. 
Between sessions only programs explicitly saved in the user program file 
are retained. 

3»2.1. Program editor 

Each editor in the CAPS system consists of interpretive tables 
specific to the language being compiled; common driving routines to 
interpret these tables; and a few routines, specific to the language, 
that are called from the interpreted tables. These tables are built 
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Figure 2a: CAPS Compiler Modules 



from assenibler-like source code written by a compiler ljiq?lementor. 

After generation, these tables are stored in coiomon where they are loculed 

into ^^ncf Ya:|^lables as needed in compiling student programs « 

Flow of control in the CAES conrpllers is shown in Figure 2a. 
The editor looks at each keypress the student enters from the temdnal. 
If the k^ indicates a text editing function, it is performed by the 
editor. If the student is entering new text, each keypress is passed 
on to the lexical analyzer. When the lexical analyzer receives a complete 
token, that token is passed on to the syntax analyzer and p€trser for 
compilation. Since each keypress is processed as it is entered, the 
compiler can give immediate error messarges when the student e[nters an 
invalid language construct. , 

While compiling new text, "Trace'^* information is stored, 
allowing the Reverse Editor to uncompile tue student's program as the 
student backs up to make a change. Occasionally the storage are»a for 
Trace information gets full. When this happens, a compression unit is 
called which removes alternate entries from the Trace table. Aftet^ 
compression is performed, the reverse editor can only back iq) to 
alternate tokens. If necessary, it will back up to the previous"* token 
and then forward compile to the current token. Ih practice, the 
compression routine may be called three or four times for a student 
program. After four calls, there is Trace information for one out of 
every l6 of the first tokens entered. Closer to the "cursor" where 
the student is working, the Trace information is available for every 
token, or at least alternate tokens. 

The module labeled "syntax analyzer" in Figure 2a is Just an 
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interface between the lexical analyzer and the parser. Its function 
is to keep track of trace information and to insure that the correct 
tables are loaded for each routine. 

3*2.2. Editor tables 

Both the scanner and the parser are table driven. The 
scanner's table is a state transition matrix derived from the regular 
grammar that defines the tokens of the programming languiage. Special 
entries in the table are provided for handling fiked-forinat languages 
and continuation "cards". (See Wilcox [ik] for more details-. ) 

The parser is a recursive descent parser with one important 
difference: a single recursive procedure can be written to recognize 
the instance of more than one non-terminal. The non-teiroinal that has 
been found by: such a procedure is passed back to the calling procedure ^ 
where it can be used to choose the next alternative. (This is equivalent 
to the separable transition diagrams of Conway [2] or Tixier's [I3] RCF 
languages). Lomet [?] has shown that such a system of procedures can 
recognize all deterministic context free languages. 

Tindall [;j.l3 has designed a language for writing the 
procedures of the parser. The language provides assembly-like commands 
for obtaining and testing input symbols and performing simple symbol 
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table operations. More' complex semantic actions are carried out by 
TUTOR procedures called from the parser. 

All of the tables for the editor can be constructed inter- 
actively using the table building modules. While under development 
the tables are stored on a disk file maintained by the table builders. 
For "public" versions of each language, the tables are assembled into 

81 

-17- 



condensed fom u in a comnon data area to be shared by^^ c^ 

teimlr^^ vritten in that langv^ With this^lsj^^ 

editor and language development can proceed independently of editor' 
use.""by ; students r ' ' ■ ■ 

The CAPS cc^^ers use "common" for pointers and tables. , ; 
shared by aii tusers^of one compiler arid "st6rage"^d^^^ 
table/sf needed by aii /indijyld user^^^ /Pew, : i* aJ0!fi:\O^^ ^"^f^®^^^^^^^^ 
are used by the coinpiler^ 

into 1500 *nc* variables in central membryii Hw 

of each may be loaded at once; ;As sho^m^n \Figi» 
- data areas in EpS carefuUy/ it|\>»s possible^; to three-area 

restriction and still get the tables in desired; Ibca^ in central 
. memory. However, the lexical and parse tiables aire ! each ffOO wrdsv!^^^^ 

and only one of them can be loaded at once. Thisvis significant^ since 

the compilers spend 8^ of their time changing the loading arraiiganent. 

Figure 2b shows the layout of these areas. . : 

3.2. 3» Edit time error 

The editor! signals an error to the student by flashing a 
box around the invalid character or symbol (See Figure 3a). While 
the box is flashing, the editor monitors the student *s key presses* 
Those which woxald move the cursor beyond the point of error are Ignored; 
the other's are processed normally, ^en one of the keys moves the 
cursor back from the point of error, the box is erased and normal 
editing is resumed. 

- A special key, labeled HELP, provides the student with 
automatically generated diagnostic assistance. The diagnostician 
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|c Symbol table 210 | 



I c Nane Table 110 | 

H • . -i 

|c Char Table 119 | 



Figure Sb : CAPS Editor Data Areas 



invoiced >by this \iey uses m algorithm siniilar to one proposed by ; 
Levy [6:], hu^^ and improved hy Tindail [12], for uigse the 

interactive environment • The actions of the diagnostician will now 
he dqplalnedv^ actions are iUustrated in the continued exaaqp 

in Figure 3. ^ 

The first mes sage generated hy the dieghostician announces 
the type of the input symbol (e.g. > keyword/ arith^ 
and states simply that in the given context it is hot peri^ 
language (Figure 3b). The syinbol class Is derived: from : ja^vf^ 
symhol table. The lemguage tables include a mapping 
in this class field into a suitable Eftglish ptoase to be used^^^^^^^^ 
and other messages. 11^,^^^ , 

After this information has been displayed^ each tine the 
student presses the HELP key again, he is shown a modification he could 
make to the text that would make the initial portion of his program 
legal. These suggested modifications are generated hy attempting to 
Insert, delete or replace a single symbol somewhere in the prefix of the 
sentential form that had been constructed by the parser up to the point 
of error. Only those modifications that lead to a successful parse all 
the way to the cursor are reported to the user (Figure 3c-3l) . 

Modifications are attempted starting at the cursor and then 
working back toward the heginnlng of the program, one symtjol at a time. 
Using this algorii;hm, each successive modification takes more time to 
compute, but the prohability of a successful parse is less. The most 
likely modifications will be suggested first. 

As the error handler works its way back from the cursor it 
backs up the parser so that it is always ready to accept input from the 
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A) FILE » WORKSPACE -^ PL/i W0RKSPACE(14-010) SPACE = 2^6 

TKfl....PROCraURE; 

DECLARE. (A,B,mD)FLOAT; 

..L1j....GET.LIST(A,B); 

.MID.=. (. (.A.+.B.) ,/.2,0. . lil 

B) FILE = WORKSPACE PL/ 1 WORKSPACE (14-010) SPACE ^ 2t\3 

TEST PROCEDURE;^- 

......... DECLARE. (A,B,MID)FLOAT; 

..L1:....GET.LIST.(A:B); 

..... ....MID.=. (.(.A.+.B. ) ./,2.0, . g 

**********ERROR»»»»»»»*** cm) OR (0*11) TO FIX. 

This punctuation symbol is not permitted here. 

Press (MEi FOR a different suggestion. 



c) FILE WORKSPACE PL/I WORKSPACE (14-010) SPACE = 245 

rf vi^v\^y vv w v v v w y v v v 



TEST PROCEDURE; 

DECLARE. (A,BJID)FL0AT; 

.. Lis.... GET. LIST. (A,B); 

,MID.=.(.(.A.+.B.)./.2.0.. Q] 



iiiii iiiil 



^•••••••••Possible Correct ion********** sssdor giusB to fix, 

Replace I I with an arithmetic operator. 

Press (MEI FOR A DIFFERENT SUGGESTION. 



FIGURE 3. EDIT-TIME ERROR ANALYSIS 
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^(^;BvMiD>FU)At;:v::V'\ 
" ; Press (S® FOR it D 




feu! 



FER;ENT^;^:SUGiSESi:i6NV-''^Si^ 



E) FILE WORKSPACE : 

> ■ TEST! ; . . .PRbeEDURE;:: 

... DECLARE. (A,B,fiID)FLOAT; ^ 'r-M^^^^^^^ 

..LL:.... GET. LIST. (A>B)r :r' :\\t;jy:./^'^r 

,MID.«.(.(.A.+.B.). 0 .2.0..; 



I I I I I I I I 1 1 



'•••••••••Possible Correction*^^^^^^*^^ CEsi or caa to fix. 

lNSERr_"r ^NJ"??!!! OpE^ _ . L^^^^^ 

PrESS(hKDFOR a different SUGGEST ion;, ^^^^^ V 



F) FILE- WORKSPACE ' PL/I WORKSPACE(l^-Olb) SPACE f 2^5 

TEST:.;.;PROCEDURE; ^ ' ''^^B-'-^-^W'^^ 

DECLARE. (A,B,MID)FL0AT; v ; : 

..LL:.... GET. LIST. (A,B); . 

MIEi.-.CX.A.+.B.H ./.2.O..; ■/V'yrr/-]:^^^^^^^^ 

••••••••••Possible Corr ect io n*^^*^^^^^^ • mtitiGR/imgi iQ fix. 

■ InSERTT'T' , IN 'FRONT OfEZI. . ■ ■ v''?;'^ '^'^^ 

Press CEI2) FOR A DIFFERENT suggestion. 

FIGURE 3. EDitrTIME ERROR ANALYSIS 
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FILE - WORKSPACE Pl/I W0RK$P4CE(ltt-Q10) SPACE " 2^5 



p w w , $ w i \ 

E; " 



E$Tu..vPROCEDURE; 
r. .7. . . .DECLARE. (A,B JID)FLOAT; 
.Ll!..,. GET. LIST. (A,B)i 
........ HID.-. (;(.AS^)'/^'^ 

••^♦^^•POSSIBLE CORRECtlON*********'\ (ES)OR tt® tO FIX. 

Insert iFRWTjjfC^.^ 

^ Press (m for a d iffereht suse^sf ioh . 

FlLE - WbRI« ^ F^E^^^^^^ -^^^^^^^ P 

fESfi....P!^(:iEDUREj - \- 

DECLARE. (A,B,MID)FLOAT; 

..Q 6ET.LIST.(A,B)i 

HID.-.(.B.A.+.B.)./.2.0..i 

« •»».. . * 

••••••••••Possible Correction'*^****^* CES) or 6® to fix. 

RepuceIZD^wjjh^an aritwicti£ operator L - - 

Press mB ^or a different suggestion. 

FILE ^ wftP K^cg . PL/I WORKSPACE(ltt-QlO). SPACE - 2^5 



IF V 



TEST:.... PROCEDURE; 

DECLARE. (A,BJID)FLOAT; 

..Q:.... GET. LIST. (A,B); 

MID.-.(.B.A.+.B.)./.2.0..i 

•••••••••♦Possible Correct ion********** CES) or to fix. 

REPLACECDjIITOj^NJkRIT^ . 

Press CaBBD ''OR A DIFFERENT SUGGESTION. 

FIGURE S. EDIT-TIME ERROR ANALYSIS 



J) FILE ■ WORKSPACE PL/1 WORK$PACE(ia-010) SPACE ■ 2^5 / 

TEST PROCEDURE; 

DECLARE7(A,B,hID)FL0AT; 

..Q GET. LIST. (,B); 

........ .NID.«.g. C.A.-i-.B. ) ./.2.O. 

•••••••••♦Possible Correction*^**^^^^^* (ES)or @si to fix. 

Replace CDvnjHH^Jjur^RioujL^ 

Press For A DiFFEREirr moEST^^^ 

Press mm to see a legal muheric built-in function 



k) file - workspace PL/I WORKSPACE (1^010) SPACE ' 245 

TEST PROCEDURE; 

i DECLARE. (A,B,MID)FLOAT; 

..Q SET. LIST. (A,B); 

mD.«. 0.(.A.+.B.)./.2.O..; 

••••••••••Possible correction^^**^^^*^^ to fix* 

Replace [ZDwith a nuheric built-in function_*AB^ 

Press cffi) for.a different suggestion.! ^^^^^^ 

Press see) to see a legal numeric BUltlTr in FUNCTIONi 



L) FILE * WORKSPACE PL/I WORKSPACEC^-OID)- SPAC^^^ 

_TEST:.... PROCEDURE; 
^?7rr77rr7BEGyU£*IA*liMip)FL0AT; 

..Q GET. LIST. (A,B); ""^^ — — 

MID.-. g.(.A.+.B.)./.2.0..; 

••••••••••Possible Correct lON^^^^^^^^^^ (eS) or esu to fix. 

Remove j I from the program. 

FIGURE 3. EDIT-TIME ERROR ANALYSIS 
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point of modification. The syxnbols selected for insertion and replacement 
are those symbols that vould be accepted by the parser at the point of 
modification. This information is iidmediately available from the parser's 
table. 

The syxnbols that oaxi be used as a modification are not only 
the terminals of the language^ but include the non- terminals as veil. 
" (Non-terminals become procedures in the recursive descent parse.) This 
allows the CAPS error handler to communicate in general terms 8uch as 
"expressions", "statements" and, aa in Figure 3f> *^uilt -in functions". 
The more conventional automatic schjemes are restricted to a vpcabul^^ 
of terminal symbols. The phriase used for each hbn-terminal 
by the language implementor. Note that as the recursive descent-ptfrser 
is backed up it returns to procedure levels closer and closer to 
procedure level of the sentence^synibol and thus the error hah 
gives suggestions for local modifications and then> only vhen these fail, 
will it suggest more global modifications. 

Whenever a non- terminal is involved in a modification, the 
student is given the opportunity to' see what a non-terminal of that 
type looks like (Figures 3j & 3k). Thus whenever a general modification 
is suggested, more detailed suggestions are available if needed. 
Currently he is shown only the first symbol of a non-teitainal - just 
to get him started - but it would be possible with conlsiderabie effort 
to display the entire non-terminal in some convenient form such as BNF. 

3*2*h. Execution 

Execution of the student's program is pefrormed by interpreting 
the internal representation produced by the editor. The internal 
representation is just a tokenized fprm^ of the program, laid down in 



the same order as it appears on the screen, IneludULng .s 

and coiBunents* This internal representation clearlyr favors the editor, 

but there is good reason for thia^. 

one": copy of each student *s program and iiliereas.^r^^ can . =1.. 

deteriorate at run-time withqut,4oing,Jtus^ is essential 

during editing. In a secisey exprading t£e.;<^^^ aqale^iia; 

actually benefici thlat-i^ ; :otjtibj^ ' 
relative efficiencies 

^ - ••. . ; . • • , . : . ; - • 

S^^ 

^ap reparai^ 
buzHiensome task - 

;For:::eai5h^ .^new?i^iai^^ 
must be desi^efS^^^^^ 
wrk vith the same form 
: ajmtool tfifcle structured , many^iM r 
interpreters* So the interpreters; 
but they are a long way from being stable toivenSLJ^ 

For the most |>art/ the features vpr^ 
depend on the progrannning language,: btxt one thing th^:m.st^' i^^ 
is a trace facility. Two levels of trace are p^^ 
only, and flow-of-cqntrol with variable 
outlines the keyword of each statement as it is executed^^^^^^^ 
tirace displays the new; each lime:dt is cl^^ 

Students are encouraged to run with flow 4jrace enabled so that^ t^ 
see their algorithm execute and obtain a better feeling for whistt^c^ 
statement does. There is one danger, however: after wa!tchljig thieir 
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_ program execute with a flow trace, students.; often think something has 
gone wroing when it executes so quickly without trace - even thotigh the 
output produced is identical.* 

3>2>5» Execution-time error handling 

the programmer i s on hand and can interact with the system to debug • 
his prdgram; In C5APS, with its ; e^ 
- goal is to discuss with the student the hattire bif the error, obtain 
from him information about specific sections of hla^^p^^ 
supposed to do, and finally to siiggest ctieu^ 

would prevent future occurrences of the error. An additio^hai g^ 
to perform these tasks in an easy-to-foilowy orderly man 
(l) the novice programmer does not become ccnftised while ^^^^ 
system and (2) he eventually learns how to debtig progranis tiy^ 

In CAPS, the interactive debugging session is M 
system and not by the student. This is essential because the beginning 
programmer does not know what questions to ask; he does not know how to 
debug. An added benefit of this is that the student does not have to 
learn to command language for ^ the debugging package. 

The CAK run-'time error analyzer [1^] is given control when' 

7""^--, ■ ■ ; . ' . .„ .. .. , ■ 

one of the interpreters has detected ^aI^obyiqus anomaly ih ^ihe^i^ 

environment - a zero divisor or a subscript out of r€mge, To^ 

The task of the error , cmcQyzer is to help the student lociat e the caus e 

of this anomaly. : : ; i 

In general an anomaly will be caused either 
assigning values to some of the variables involved or by an error in , 
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setting up storage for one of these variables* To locate the cause of 
the error in the first case ^ -the analyzer must reverse-execute the program^ 
searching back through the histoxy of execution to find out \Axere and 
how those variables were assigned the troublesome values* Similarly^ 
in the second case^^ the aneiLyser locates the cause by reverse-executing 
the program to the point of Instantiation of the variable and then searching 
back througlh the history of execution to find where and how the variables 
that controlled the instantiation were assigned the troublesome values* 

In both cases the analyzer engages in a discussion with the^ 
student while internally it is reverse-executing his program from the 
point of error look^^for assignments to one of the troublesooae variables 
involved In the erro^T Once such an assignment has been found, it^is 
shown to the student* If this does not help the student locate the 
cause of his error, the error analyzer then looks at the expression 
that computed the troublesome value to i^e. if any suspicious conditions 
are present that could be the cause of the error* Whenever a possible 
cause is located an appropriate discussion about the condition is initiated* 

For this analysis, a common : misconception tabie : is used* Tlie 
table contains izrformation about situations in the lang^ are 
potential trouble spots as well as tenqE>Iates for the actual discussions 
that should be Initiated if the situation is recognized* OiscUssions 
located in the table are referenced during expression analysis by an - 
(operator, value) pair In the following way: As ah expresislbn is being 
ancdyzed for the cause of the error, each operator md the value it 
returned are looked up in the table* If the pair is present^ the discussion 
corresponding to it is initiated, otherwise analysis continues* The 
analysis of expressions is done by a pre-order traversal of the 



is :.moM • tha^^ .one .conawtt '^'^^ '•'JPI^;^ 
i: '1 ■ • 5 is discussed if l^st ; ; Then siiaiB^ 

.' ^ C: ijiellj/'l^^ "inneir ones »e:. (Uscuss^d^Ho 'piw ''Wie '.'^^ jebo^ 
'\-^''y]P' -^''& altufiWon/ . •that;theM^^^ 

;; . i^i?* bprrespcmdlng common ;mi^ ^.^^ 

.-'D^sconce^ 

aviA- '""^iirtM A*^4t- AW ■'liAine-; ^bittto!ei- '.discus 



If no ^ Common ■ tteiidiscusS^i^^ 



' . •■.'TOvdsa';^ cause-' 6f thei^ir^ 
' varii^ in ^ the ^^e^ that' ^ di>^ iicyt {: seoa/^^ 

; .^^S- ■ ■ ^- : ■ • : ^ / . ^ ; -..f ■ . ' - -;> ; [:-k-'.{>J^::^^^ 

changed by the expression ^ust analyz^^^^ 
. trcnjblesdne^ variables and those:'. vafialil^ 
V :'added.. -RieveMe-execut is" thett;resini«<fc 

^ O^ ■[., [. \ -.y^0:: '^^^--^^ 

■ "V;* . NOte that: idle selectidni:'^^^^ 
: ne^cepsaiy , b^ it does perinjt otaculjar :p^^ - v 

' - -rmakes^' it clearer ^ to' the: student'^how iiti^ wca^^ i J:B^kB^-. 

Error analysis terminabea^^ 
,..'V;ariaer;- :(1) ttie-last- trciiblesome..,variabIe:;^^ 
::Tconstant ; (The'' cbhstctnt^^is ^Jjicorrect.) ^or/ 

(The viatlue iEbirt ; is. incbrrectvlri (2) ttie ; s^^ 

' ' The; ecaaple in Figure U should demcmstrate the features of , : ' '.T 
-^-S the .run-time error emaly^is. . •Ass«mlng:vt^^ -i f^^vli^l 



PL/I px^ogram shown> he vould receive the execution error as indicated 
at the bottom of Figure Ifa. If he requests help, he receives the display 
in Figure Ul> showing him vhere the variable A yas assig^^ . 
value 0. When the student reqpiests more help, the expressi<m c<m^ 
the value assigned to A is analyzed, Ih this case (flx^ :^y:iC^^ 
appears^in the common misconception jtabre' i^^ 
shown in Figure Uc is presented to . the st^ldent•^^^;; ^^ 

more help, the pre-order traversal continues As [tl^ '}}^'-.:'i<^^. 
programmer is asked about the reasonkbliii^ of y«^^ 
(Figure Ud). If the student asserts that i± is r^ 
the value 8, the ajMOyzer must respond with the "^oi^l^ 
Since there are no more troiiblesbme variables ^^t^^^ 
for help are answered with the message shewn at the bott^ 
If the student admits that 8 is an unreeuaonable val^ 
that gave A that value is located and shown to the student (Figiire^^ 
Since this is an input statement involving the only trotiblesome value, 
the analyzer terminates with the conclusion shown in Figure 4h« ^ . ^ 

The run-time error analysis algorithm is essentially:^ langu^ 
independent. The language dependent information is contained^ 
common misconception table. In addition, . each language inte^^ 
generate a history of execution in a standard form (similM to that 
by Zelkowitz [15] ) and provide a routine to constinict 
from the internal represt=r*'^ation of a statement. 

"'3.3. Eviitl ue tion , ^ ,;■ ■ :• ^ u^^^^ 

OAFS is an experiment in extending the concept of pedai^ogic 
progranning syistems to the interactive environment* The logical setting 



..Mi,'> . ; 



} ^ PL/I EXECUTION 

"-exampliyL procedure; OPT^ 

:':v;....i.,DECLA^^^ 
..^/.:../;6ETiLIST.(A); 

A4/Ai~^ ~ 

..........END; 



Execution Error: DIViSIONhBf^ERO. * : 

f RESS^^^) -FOR ERROR analysis; fiiaD FOR LIST OF 0|HER QPTIONSi. 
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a) Detection OF ERROR, (Student presses ppP. key,) : 
FIdURE 4 : EXECUTION -TIME ERROR i^NALY^l3 



EXECUTION-TIME ERROR ANALYSIS 



EXAMPL2 ! . PROGEDUREiX)l?TlONS (MAIN) ; 
.... . V M DEGLARE^^ i F IXED; : [ 



. v.. V...; get; list; CA); 




POSmt)N^^OF€RROR^ 



THIS' istATEMENT GAVE A AN^ 



ISSil^lP: Ep IT .YQURLiyifiGR 
CSSI TO • SEE HOW EXECUTION FLOWED FROM HERE TO ERROR . 



b) Assignment THAT lead. TO error, (pRES^ 
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FI^RE4. EXECUTIOnIt^ 



EXECUTION-TIME ERROR ANALYSIS 



EXAMPlil.PRdCEDURE.OPTIO 
;;..M.i.DE^ 

. ;»6ETrLISTr(A); 



• I I I • 




This DIVISION operation performed her€~wasv integer 
division. In this case/ the resulting value was 
zero because the numerator was less than the 
denominator. 



Press rnon to edit your program; cmE) for more help. 



c) Possible cause of unexpected result. ^ (Presses' 
FIGURE 4. EXECUf ION-TIME ERR^^^ 
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BECilTlON-TIME^TO 



• tit • • VtNi^^ 

• i, iX*" • • • ENJI# : 



POSmONiRiERRdR- 



THIS -StAttWENT 6/WE A^AN IN 



Does a value of 8 look reasonable^ p^^^^^^ 

Answer NO :yNLEss you are absolutely rositive ^ 



Type Y OR n. 
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b) Request FOR input from STUDENti^ ^; (Presses '^V^^^ ; 



FIGURE 4. EXECUTION-TIME -ERTORtA^ 
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ERROR ANALVSiSl 



.v:^>:|iiiaii0S» 




On the. assumption that all dFv^ouR j?revi6^^^^ 



ALL - dF'V^OUR-'"'i?REV;i;GU.S"v^:^ ^ ■ ■ 



RESPONSES J WERE TRUE, THE INDtCATEp 
eONTAINS A. LOGIC ERROR.^ THESEASbNlFOfci^^^ 



v^tp*!* i> YOU SAX THAT ALL VAR I /I^BLES - COpAI NED WI 

!;:tr..^r;;iT.;^coNTA 

feiVi^RIABLElMA^^ :cOKTA:IN.;THE:?RES0l|TrNG^^^^^ 




e) iJl splay: GIYEN WftEN stupjENT TYPES - 
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EXECUTION-TIME :ERIilC(R^fttl»tS 



mm 



' THI.S^ STATEMENT: GAVE A AN INCORRECT 



On the assumption that alL' of your >rev to^^^^ 



RESPONSES WERE JRUE. THE INDICATED EXPRESSION 
CONTA I NS Vi; LOG I C TERROR' , 



The >SEAS.0N-F0R>^THIS :IS 



*R^sioN^^ 'v : -m^^^mMmSi^^^ ■ 



THAT^tXPU^lSAY THAT ALL VARIABLESivCOIJiXMNEb:^^^ 



IT CONTAIN CORRECT' YA.LUESJ;?rBUT^^THEtliE5tlte?i:0N:i 
VARIA^LEiMi^ HOT 
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F.) All help exhauster. (KELP kfev no Lor^ER Afci^VEft"^'; ^ 
'fl<5URE/if^;,.E)<ECMTION^jIIIME''rE;R90l!tl^^ 



While no one has yet developed a universally accepted 
technique for organizing a body of knowledge, there is some consensus 
that a useful point of view is to model knowledge of a subject as a 
network built of concepts and relations. Hence the data structure 
for the GUIDE r ocept space is simply an abstract graph w^ 
nodes of the graph are concepts and the arcs are relations between 
concepts. The choice of this extremely simple yet powerful model was 
fully vindicated when it was put to use. It was found to be adequate 
to ixscorporate the synony^^j dictionary, the hierarchical classification 
scheme, and the term clusters which were originally proposed as separate 
components of the concept space. Also, it serves as a keyword index 
(holding all keywords which have been attached to lessons) and a 
thesaurus (holding all subject-matter terms known to the system and 
showlitg-how they are related). Furthermore, when it was desired to 
anJjidexjto the library of lessons, the concept space eJ.ready 
"^provided, the necessaS^'^-^eGBs^^ And finlEaiyj-itjjr^ 

to specify the structure^r a courae by means^^l 
listing of lessons, again simply by utilizing-^he' 
in the structure of the concept space. ^ 

It should be emphasized that the word "concept" is used rather 
loosely. Any word or phrase which is the name of a node in the concept 
space is called a "concept". Similarly, the relations used in the 
concept space were chosen so as to be intuitively clear to the user. 
These relations could be readily extended if it proved desirable to do 
so, or if a universally accepted set of relations among topics were to evolve. 
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The relations currently in use are: "generic-specific, cpntainor- 
containee (\rtiicii iniposes an index structure on the lesson library), related 
(for concepts related, but not by one of the more specific categories), 
synonym, owner-meniber, ani prerequisite-sequel. 

A'concexrt record, then, consists of the concept name, a list of 
lessons in vhich the concept is a keyword, and a list of relationships 
with other concepts. 

Word to term translator 

The word to term translator accomplishes the task q'j! transforming 
a sequence of input words into a sequence of terms recognized, by the 
system. This is done by extracting the leftaaost word of an input request, 
applying a hash function to that word, and looking in the hash table in 
the appropriate location. Entries are arranged in the hash tabe in such 
a way that it is possible to extract from the original request the longest 
'possible substring which matches a term known to the system. The progress 
of the transudation process is communicated to the student by underlining 
each term when it is found in the term dictionary. 




processing. The translation is not based on an elaborate linguistic 
analysis; rather, the translator^ searches through a space of partial 
meanings determined by an analysis of the domain of discourse. The 
translator's approach to dealing with natural language can be likened 



to a person who is hard of hearing. Even though such a person does not 
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efjiM^ all the vords scmeoM speaks, he caih u^^ the 

mWaning. of a sentence 6n tte of his Imowrledge of general topic 

of cbnversatibri^fimd the f ew^^^w^ he did hear clearly (including the 
jordering and context of those vords); 

. .. .1 The^^ jas.ed. JU3..*he :t3^^^ 

ot. a nondetermlnistic finite state autoinaton (Figure 2);/:; Ba^ on the 
current state Of the autoinaton emd the input class^as-s 
is encountered, the appropriat in thb state^^ 



addition to "Im made to ^fie intemediate r^ 

and also establishes the ne3^ state to be entered by the ^^^^^^^^^ 

state of the automaton can be associated vlth a partifl^^^^^^ 



the request. This understanding is based on that portion ^of the^^^x^^ 
which has been analyzed up to that point. If the trauislator reaches a ' 
dead end in Its search for a inecuiingful interpretation of the re quests 
it backs up to the previous term and looks in the state table for an 
eatemative interpretation. This process is continued Imtii 8^ choices 
are exhausted or a consistent interpretation has been found; /^^ this 
approach eliminates the need for storing a grammar of Baglish;X(^ 
considerable amoxmt of memory) and allows the tranisiator to hflwdle 
uiigraimnaticeQ. or partially imderstood inputs. 

The intermediate representation is a nesting :b^^^ 



to routines in the request pro5(Bi5T5orr-^The_possible functions are shown 

in Figure 3. The basic idea is that most requests have a sim^ 

one section indicating the type of information desired,: and the other a 

series of specifications limiting the domain of interest. The ft^ 

can then be divided into two groups: those specifying a particular lesson 
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dead state 



accepting states 



Figure 2: The State Diagram of the Non-deterministic 
Automaton 



or set of lessons; and those for particular types of information, having 
as arguments a specif ication function, or a neisting of type and specificatioii 
functions. This is discussed more fully in a Ih.D. thesis ; by Fradels [2].. 

- . Baraphraser ^ '. , 

The paraphraser produces a paraphrase of the original request 

based on the intermediate representation produced by the English translator, 

. ' / 

allowing the student to confirm whether his request has been properly 

» ■ , • . . . , • ■ ' ' - , ... 

understood by the system. If so, he can proceed to. the response. If not, 
he can immediately rephrase )xis request. (Also, in many cases, he can 
deduce what caused the system to misunderstand his request.) 

km6m Request processor 

The request processor accomplishes the task of transforming 
the intermediate representation of a request into a specification of 
the type of response to be generated. By analogy with the output of 
the English translator, this specification has been called the "intermediate 
form of the response". 

Several simple heiiristics are used in the request processor, 
based on the principle of determining as quickly as possible which area 
of the database contains the answer to the original request, and what 
possible response of the system will display that area of the database. 
In some cases, the request processor simply Indicates to the response 
generator the area of the database to be displayed (for example, the 
term number of a course record). In other cases, the request processor 
assembles some data frcM the database and passes that information to the 
response generator (for example, a list of term numbers or lesson records 
^vAiich match a given specification)* 



Specification functions: 

These functions rettirn a set of lessons depending on the value 
of their variables. 

LN : has two arguments^ a course name and a 

lesson name. It returns the lesson defined 
by these. 

LS : returns the set of lessons which are defined 
by characteristics other than thQir names. 
These characteristics might be a'^Boolean list 
of keywords, a type (lesson, exam), a course 
to which they belong, an author name, a level 
of difficulty, a sequence specification, a time 
period, or other specifications, such as whether 
the lessons have already been taken or not. Any 
of these characteristics can b$ specified or negated. 



Information type functions: 



TF : tests if the set is empty or not 

AB : returns abstracts of its elements 

AN : returns names of author of its elements 

NB : returns the size of the set 

GER : returns the grade required by the instructor 

GO : returns the grade obtained by the student 

TS : returns the schedule to achieve relative to the elements 
of the set , 

TT : returns the schedule achieved by . the student 

PQ : returns the set of lessons which are prerequisite to the 
argument set 

SQ : retxims the set of lessons which are sequels to the 
argument set 

SI ; returns the set of lessons which are similar to the 
argument set 

BE : has two etrguments, a lesson and a set. Tests If the 
lesson belongs to the set. 



Figure 3. Request processing functions 
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U.7« Response generator 

"^^^^^^^ T^^ accomplishes the tSiSk of transforming 

the intermediate form of the response into a display ^ich can be 
presented to the user. There are four basic types of response "Which 
vill be discussed belov; list of lessons j,_ r^^ _ _ 

display^ and feedback message. 

U»7»l» List of lessons 

The response to a large number of requests presented to the 
GUIDE is a list of lessons matching a given specification* For this 
response; the response generator simply produces a listing of the name 
and abstract of the lessons which have been' retrieved^^ 

U.7#2» Record display 

Record displays are generated for lesson^ course^ and student 
records* Ih the original design of the response generator^ it vas intended 
that most requests would receive a prose response* As an intermediate 
step in the development toward that end, it was decided to display the _ 
entire record which contained the piece of information wtdchj^^d 
requested._^ This - aj^roach to' the proved to be so succiessful 

that implementation of the prose approGLch was abandoned* 

With this approach, we anticipate in advance a whole, sequence of 
potenti€LL questions (hence "reducing total CPU useige)^ >fi^ properly 
answer a large class of poorly-phrased questions* This allows the student 
to type shorter requests and still obtain the desired iiiformation* 



U,7«3« Graphic displays 

One of the challenging reseetrch tasks In Implementing the 
GUIDB vas the development of an effective means of communicating the 
straicture and content of the concept space. This network possesses, a 
very rich structure of Interrelatlons^ps vfaloh is <!ilfflc^lt describe. 
Fortunately, the HATO termlneil provides a graphics capability which. ' ■ 

helped solve that problem. The GUDJE utilizes^t ; 

of graphical displays to help present different points of view of ^ t^^ 
concept space: the nelgpiborhood, hler€u?chicaly and mixed i^^ ^ 

The neighborhood display shows the concepts which lie: In the 
immediate neighborhood of a given concept. Figure U shows a san^e 
neighborhood display. The first circle of nodes shews -the "first 
generation" of concepts— those that are directly related to the central 
concept of the display. The secondary circles of nodes -show the "sm 
generation" of concepts— those that are directly related to the first 
generation concepts (and hence are two generations away from thexentral 
concept). Exploration of the concept spax^e can be ax^complished by 
requesting successive displays where the central node in each new^ 
display has been selected from the first or second generation of the 
previous display. 

Whereas the neighborhood display gives a sense of "dlstaiace" 
in the concept space, the hierarchical display Imparts a sense of "direction 
Figure 5 shows a sample hierarchical display. Basically, the hierarchical 
mode enables one to traverse the classification tree, pursuing topics by 
narrowing or expanding one's scope of Interest. The sense of direction 
imparted by the hierarchical mode is how "high" or "low" a given concept 
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Figure k: ' Sainple Concept Space Neighborhood Display 
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O denotes concepts 

^ denotes Concepts used as keywords 
what next? (active keys. l,C|S»e; h^n^m; r«o; 



BACKl for index) 



Figure 3i Sample Concept Space Hierarchical Display 
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1 do loop 
2*plldo 
3«fortdo 
4* loops 
-5-Pb/I-L«ngu«ge — * 

6 loop 

7 do statement 
e iieratiort 

9 pi I 

10 FORTRfti Lanftuage 

11 fortran 

15 Language Indepen 
dent Progir aiming 



n denotes lessons 

^ denotes concepts used as keyMiords 
what next? (active keys: l|C|S,e; h,ntm; r«o; BRa<l for index) 



Figure 6 Sample Cpncept Space Mixed Mode Display 
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^^^}^b:4jx relfictive to ther 

i#;;;oif ihb 

The mixed mode display vaa^i^^ 
^graphical presentation of the set of lessons licdc^^ cu:e attaxshed '^^^^ a 

™~given-.conc©pfb^- It is :c€ajLed i?mixed".-since both^le&sons^ and-concepts^^^^ 

appear in the display. An exanrple of such a display is shown in 
Figure 6 • An interesting interpretation of th:U display is thai in a 
very real sense^ a lesson can he viewed as a relation in the concept 
space^ providing a link between various concepts. For example^ in 
Figure 6, the concept "do loop" has the "fortdo" relation with the 
concept "fortran," 

* — • 

**.7.^« Feedback response 

A feedback response is presented to the user asking tor 
clarification or further iziformation in three sitmtions: when an 
ambiguous term is used> when the translator can't find a valid ^ 
interpretation of the request (often caused by a significant word of 
the request not being in the database)^ or when the request processor 
does not have sufficient information to process the request (e.g* ^ no 
student record available), ' 
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t ;;v > exaavi^ystem - Xl> R4 WhltloclCt Rt l^ Anderson )v y-.;:^ yv ■ 

The^^^G^ System is ; a cicra^ system 

fy>r tiie cbnsH^ru exa^natioris# Sincse all tM ^K; 



associated with examinations (from exain vriiiiig tlurough 
resiilts) are hahdled^^l^ In ^toe.s^ 

System of fers ^many adv^ oyex^^'wrl^en^ ^e^^ iiiclui^ 
a consldereible savings in time and expens^^^ and ^ 

grading exams; exam seciurity, provided by ^t^^ each student 

receives slightly different questions; cohsifstm^^ exan 
grading; the capability of edlowing each studra scorM^^^^^^^^^ 
correct answers on his excua immediately isuf^er he finishes it; an 
immediate availability of a complete analysis of exam restiLts after a 
class finishes an exiam. 

The operation of the Generative-Exam System is described in 
detail iii one document (l), and the development and evaluations of the 
system are described in detail in another (2). Some of the major 
aspects of this project are outlined below. 

5«2« System organization 

The exam system differentiates between two kinds of users *- 
student and instructor. An instructor has access to both student and 
instructor options^ while all other users have access to the student 
options only. 

Figure 1 is a block diagram of the major components of the 
Generative Exam System. All users enter the system through v the Monitor^ 
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Figure 1: Block Diagram of the Major Components of 
the Generative Exam System - 
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and on Initial entry are allocated a record in the Student Records data 
^ase and a permanent storage area for their work in the Student Escams 
data area. Instructors vrite exams in the Exam Writing section by 
writing problem specifications for each desired problem generator/grader 
(pg/g). This set of problem specifications is assembled into au exam 
specification and stored in the Exam Specs data area. When a student 
takes an exam (in the Exam Administration section the appropriate exam 
specification is transferred from the Exam Specs data base to the 
student's permanent storage area in the Student Exams data base. The 
same area is used to record his work as he changes from problem to 
problem. Instructors may review exam results in the Exam Statistics 
septic . 

The heart of the sys-cem is the set of pg/g modules which 
produce examination problems. Each pg/g is an independent module which 
handles all aspects of one problem except data storage. These functions 
of a pg/g include guiding an instructor through the process- of- writing 
problem specifications, generating problems (under the constraints of the 
problem specifications), administering problems to students, and reviewing 
problems with students after their exam. 

Since each student ^s problems are generated as he takes his 
exam, there is no pre-test security problem. The generation schemes 
used by the problem generator/graders are designed to operate under 
time and storage space constraints so that delays and distractions to 
the student are ayoided. The generation schemes produce a large number 
of similar problems by randomly generating niunbers and character strings 
and assembling problem pieces into complete problem structures. Some 
pg/g's have the capability of generating problems at different specified 
levels of difficulty in their subject areas. 

^5- 



The problem generator/graders employ grading schemes^ ^ 
anard credit for partially correct responses by checking responses for 
variants of the correct answer or by grading the correctness of one 
response on the assumption that the previous response In that problem 
Is correct* An example of the former grading scheme Is found In the 
FORTRAN expressions pg/g* If the correct answer to an expression vere 
"-U5*0", the pg/g would award partial credit for the responses '^lf5«0", 
"U?", or The DO-loop pg/g uses the second grading scheme mentioned 

above. The response given by a student for each Iteration of the DO-looj^ 
Is compared to the correct answer for that Iteration and to the answer 
calculated from. the student's previous response. Full credit. Is awarded 
If the response agrees with either answer. 

5#3« Experimental results 

Two experiments were conducted to evaluate the Oeneratlv Exam 
System. In each experiment, subject a w^re administered an exam on 
PIATO and a written exam. The coefficients for the FIATO exam scores 
correlated with the written exam scores averaged •61f In one experiment 
and .60 In the other. Assuming that the written exams gave valid 
measurements of each student's knowledge^ these results suggest that 
exams In the Generative Exam System are as effective at evaluating 
students as written exams. 

The experiments also studied the tailored style examination. 

In a tailored exam, the difficulty levels of the problems are altered as 
the student works throu£^ the exam In an attempt to match the problem 
difficulty level to the student's level of knowledge. This approach . 
should more accurately measure the extent of a student's knowledge and 
make this measurement In less time and with less frustration to the student 
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tim the tradlt 1cm style exaadnatlcm. A be useful 

In criterion-referenced grading sv^h bm^ courses. 

In the. experiinents conducte^^^^^^^^ 
System, some subjects took tailored FIATO /exams and other subjectr took 
regular PIATO ^ams. (Regular HATO exams are veiy sladlar to yrltten 
exams.) The coefficients for the PLATO exam scores correlated vlth the 
vrltten exam scores were higher for the group of subjects Vho took 
tailored exams than any other HATO exam group (.83 for the tailored 
subjiBcts versus an average of .39 for the subjects Vwho took regular 
exams In one experiment, and .68 for the tailored subjects versus an 
average of .33 for the subjects who took regular exams iq the other 
experiment). These results indicate that the tailored exam idea is 
at least as effective in evaluating students as regular style exams. 
However the implementation of the tailored exam in the Oenerative Exam 

System was inefficient in terms of time (tailored s\ibjects spent an 

J]' 

average of ^0.32 minutes on their exam as opposed to an average of 
31.78 minutes for the other subjects) and was unpopular (as indicated 
by questionnaire results). Improvements to the Generative Exam System 
which could make tailoring more efficient and less unpopular have been 
planned. 

The studies conducted with the Generative Ebcam System suggest 
that interactive' exams are useful and effective in evaluating students 
and merit continr.ed research, especially in the areas of problem 
generation and grading and tailored exams. 

5. If. The quiz system 

In an effort related to the exam system, a special quiz system 
has been developed which enables presentation of a criterion-referenced 
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quiz following a SLATO coo^uter science lesson. • Ztoslgned and Implemented 
by R« !• Anderson ftom a concept proposed by R. G« N6stanelll| the system 
is Intended 1) to provide a student taking a ELATO coniputer science lesson 
with both a means to assess how well he or she learned the material that 
the lesson is Intended to cover and a tool to aid In learning the topic 
at hand, and 2) to provide members of the ACSES staff with a means to 
assess how effective and thorougb a PLATO coniputer science lesson Is at 
teaching Its topic. 

5.U.I. Quiz system operation 

The £fystem consists of a quiz system monitor and a pool of 
PIATO quizzes available for administration to students at the conclusions 
of individual computer science lessons. Each quiz has been designed to 
pertain to seme well-defined topic within the computer science field, 
and each quiz question has been selected by the quiz author to test 
pertinent details of the topic's content. Incorporated into each quiz 

is a data collection facility to recoxd students* question responses. 

/ 

Access to the pool of quizzes is provided, via the quiz system 
monitor, to Instructors who wish to use the quiz system. The system 
monitor allows such instructors to select a quiz of the desired content 
area, interactively design the quiz to best suit the particular lesson *s 
needs, view the qpiiz exactly as a student will, and finally "attach'' the 
quiz to the chosen Instructional lesson. This latter task requires minor 
changes be made to the code of the instructional lesson to enable an 
Mterface rlth the quiz. Tl:>ese changes are clearly outlined to the 
uyer when quiz attaciiment is arranged. 
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Figure 2 illustrates how the instructional lesson/quiz interface 
operates. At the time a quiz is to be administered to a student, control 
is transferred from the instmctional lesson to a quiz system program 
that functions as a lixik to the quiz (arrow A). This program determines 
which quiz of those available in the system is to be presented for this 
partictilar Instructional lesson, and control is transferred to it (arrow B) 

Interaction between the system* program and the lesson that 
produces the quiz occurs at various points during quiz administration 
(arrow C). Since all quizzes are designed with a similar stmicture, most 
aspects of student-quiz interaction are uniform across quizzes. The basic 
sequence is as follows: 

1) Once the quiz system program verifies the existence of a quiz 
for a particular instructional lesson, a page detailing the 
quiz's purpose is presented to the student. 

2) Quiz questions are then administered; questions may ba sK.lMTjed 
if the student so desires, and all questions may rewisv- red 
in ease an error was made. 

3) When the student decides that his or her attempt at t^o *iz 
is complete, he or she may advance to the presenter a of the 
corrected quiz, which is accompanied by clarifying explanations. 

k) Lastly, the student is informed of his or her final quiz scores 
as well as the average score received by others in his or her 
course. 

Following a review of the corrected quiz, a studrtit is returned to the 
system program a final tlxce (arrow D) Vhere return to the appropriate 
instructional lesson ie provided (arrow E). 
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Figure 2: Simplified View of Interactions 
During Quiz Administration 
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Once a quiz has been attached to an instructional lesson, 
each student taking that lesson will also take the quiz, and be informed 
of the average score obtained on the qrdz by other members of his or her 
course. Each instructor will have accerot via the quiz system monitor, 
to data that is accumulated for each quiz question and will be informed 
of both the average quiz scores and the average amount of time needed for 
taking the quiz for all courses that have used the associated lesson. 
Each authorized ACSES staff member will also have access to the data 
accumulated for each quiz question, to facilitate analysis of the 
effectiveness and completeness of both the instructional lesson and the 
quiz. 

When course instruction is completed, instructors detach the 
quizzes from the associated lessons. This procedure, which can be accomplished 
through the quiz system monitor, clears the accxMulated course data. During 
the time that a quiz is detached from an instructional lesson, no reversal 
of the quiz-accomodating lesson alterations made earlier are necessairy. 
transfer of control to the quiz system program to administer the quiz 
(arrow A of Figure 2) will simply result in the display of the message: 
"No quiz currently exists for this lesson". Control is then immediately 
returned to the instructional lesson (arrow E). 

5.U.2. Past experience and current status 

Currently, only quizzes pertaining to selected topics of the 
FORTRAN programming language have been implemented. As the first quizzes 
of the system, these were all designed and developed from objectives used 
to develop the existing FORTRAN instructional lessons; thus instructors 
had very little opportunity to manipulate the quiz design to satisfy other 
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lessons* needs. More quizzes are being developed^ however, and ebclstlng 
quizzes are continually being improved. The availability of a quiz that 
suits a user's lesson* s needs may be investigated via entrance into the 
quiz system monitor. 

The initial trieLL of the quiz system occurred during the fall 
smester of 197^> ^en a FORTRAN character manipulation quiz was presented- 
foUowlng an instructional lesson on the same topic. Q^iz question 
responses accumulated from students in an introductory conputer science 
course clearly indicated various deficiencies within the instructional 
lesson and even revealed an instructional error. The lesson was thus 
restructured^ the discovered error was corrected^ and the identified 
deficiencies vere eliminated. 

Subsequent use of the quiz system occurred during' both the 
spring semester and the summer session of 1976. Four quizzes vere presented 
following appropriate instructional lessons to students in four different 
introductory computer science courses. Preliminary analysis of data 
accumulated by these administrations indicated shortcomings both in 
instructional lessons and in quizzes. Corrective action is currently 
underway. 
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6. Automatic judging of student programs (R. L. Danielson, P. Mateti, 
W. D. Gillett) 

An automated system for instruction should be capable of 
making Judgements and providing comments on student programs, analogous 
to the role played by teaching assistants and graders in the more 
traditional means of instruction. Our efforts to provide this capability 
have resulted in two lessons which ask the student to write fairly 
sophisticated programs and attempt to Judge these programs interactively 
with respect to both correctness and good design, and a categorization 
of anomalies in beginning students' progi*ams which are detectable by 
automatic analysis routines. 

A program by R. Danielson exposes students to ^ dynamic 
example of the top-down programming process by monitoring their attempts 
to write a PL/l program for symbolic differentiation of a polynomial. 
PATTIE (Programmed Aid for Teaching Top-down programming by Interactive 
Example), mimics the action of a hximan tutor, in that she engages the 
student in an interactive dialog. Judging the correctness of student- 
suggested refinements and providing hints and comments where necessary. 
The tutor uses an AND-OR graph as a model of the stepwise refinement 
process, which student and tutor traverse together in the course of 
program development. Danielson (1975a, b) discuss this tutor in detail. 

The other lesson is a sorting laboratory and program verification 
system developed by P. Mateti. This system allows the student to write 
an arbitrary in-place sorting prof^am in a programming language with 
specially designed sorting primitives. A special interpreter then 
provides a dynamic display of the status of the array and indices during 
execution. In addition, the student may provide assertions about the 
state of the keys in the array, and the truth or falsity of these 
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assertions is indicated diiring execution. The student may submit 
completed programs to the program verification routines, vhich use 
the inductive assertion method to prove the program correct, or prove 
it incorrect and provide a counterexan^^le. . A special theorem prover,. 
vhich is highly efficient in this restricted domain, is the heart of 
the system. A ful3. description is in Mateti (I976). 

FineuLly, a study is being conducted by W. GiUett aimed at 
determining and categorizing various legcuL programming constructs vfaose 
presence in a program probably indicates a lack of understanding by a 
student (e.g., B**l/2, vhich is equivalent to B/2). The idea %8 to 
determine techniques by which such errors may be detected; the general 
approach is to use iterative analysis methods on a flow graph equivalent 
to the student's Fortran source program. An automatic program containing 
such techniques, while being unable to direct the student toward correctly 
developing a program because it isn't aware of the algorithm being 
implemented, would still be able to provide Incisive comments on Improving 
prograjn efficiency, correctness, eind understandability. 

The following subsections provide a more detailed discussion 
of these three efforts. 

6.1. PATTIE 

Top-down programming provides a means for the programmer 
to restrict the scope of the :n\ ..lem he must solve to a manageable 
level. The principal aid in this re striction of scope is the use of 
levels of abstraction. Successive refinement begins with an abstract 
description of the task to be accomplished. This task is then refined, 
that is, described as a serlun of slightly more specific tasks which, 
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when coinblned; solve the problem. Each of these tasks at this second 
lever is refined in turn, producing. a third level of task descriptions, 
and the process continues until tasks have been described in sufficient 
detail to be easily translated into prograjoming language statements. 
Task descriptions commonly employ a mixture of natural language and 
programming language statements, -which allows much of the complexity 
of the programming langiiag^ to be ignored until needed. The successive 
levels of task descriptions allow the programmer to concentrate most 
of his attention on the task he is currently refining, and yet be' sure 
of the proper integration of that task with the whole solution. 

There must be three separate aspects of a system designed to 
tutor a student about top-down programming. First, because the tutor 
must monitor the process of developing a program, it is necessary to 
provide a representation for acceptable methods of solving a problem, 
as well as acceptable completed solution programs. Second, because 
we want the student to learn something about the technique of the 
top-down programming, the tutor must have some instructional strategy 
to aid this learning, and use this strategy in Intetaoting with the 
studetit. Finally, the importance of nattiral language to the succenslve 
refinement process requires the tutor possess some natureO. lemguage 
capability sufficient to understand suggested refinements and allow them 
to be related to the knowledge of acceptable solutions. 

Let's look at each aspect in further detail, 

6.1.1. Representation of knowledge 

Any sort of problem solving activity (such as programming) 
involves reducing the original problem to one which is understood and 
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can be solved^ using some rules or problem reduction operators • Problem 
solving tutors for other subject areas (simple integration, logic theorems) 
give the student a wide range of experience, and are capable of handling 
a correspondingly wide range of both prestored and student-suggested 
problems. To accomplish this, heuristic problem solving routines, with 
capabilities similar to those of the students being tutored, are integral 
parts of the system. Such an approach is posslJ)le due to the quantitative 
nature of the subject areas. Solving problems in integration or proving 
sljuple logic theorems requires using only a small number of rules 
applicable to many problems. 

Unfortunately, in top-down programming there is no small set 
of general rtxles which can be applied in many sitw^tions. There is, 
instead, a very large number of distinct refinements which are applicable 
in only a small number of instances. This, coupled with the difficulty 
of clearly establishing a new problem state following a reduction expressed 
in natural language, led us to explicitly store knowledge about the exact 
solutions to a particular problem, and change this knowledge to allow the 
tutor to accommodate other problems. This leads to a need to represent 
a top-down solution. 

The traditional representa*Cion for the stepwise refinement 
process is a tree. The root represents the initial problem to be solved, 
leaves represent statments in the target programjilng language, and each 
intermediate node represents a subtask jn one of the levels of abstraction. 
Such a tree, however, represents only one solution; there are likely to 
be many correct solutions to any particular problem. Hence we decided 
to represent the solution knowledge as an AND-QR grah. 
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The basic idea behind an AND-OR graph is reducing a problem 1 , 
to a series of subproblems^ just as In steptdse refinement. In such 
a graphs each node represents a problemt Solving thie problem represented: 
by an AND node can be accomplished by solving all the subprobleM 
represented by the successor nodes. Solving the problem represented by 
an OR node can be r?X5Corapllshed by solving any one of the subproblems:^^^ - v 
represented by the successor nodes. The solution to the Initially glyien : 
problem (represented by the root of the graph) is successively reduded; " 
to the solution of sets of subprcblems, some of vhlch miglrt Tie ^ 
immediately recognized as being solved (LEAP nodes), others of vhiic^ 
might need further reduction. Intuitively, an QB node corresponds to 
a point in the development of a solution where a choice mMst be made 
between several (equally correct) approaches. An AND node represents > 
a point at which refinement Involves several tasks which must all be 
done to solve the problem. Figure 3. is a small portion of the AND-QR 
solution graph for the problem the tutor is currently using: developing 
a program for the symbolic differentiation, of a polynomial. 

• In order to use an AND-OR grr,jjh as the basis for. a tutor of 
successive x^efinement, several features were added. As Figure 1 
shows, branches between nodes are tagged with English phrases ("transition 
phrases") which are usually descriptions of the 'sasks represented by the 
node each branch leads to. Thus, nodes represent siibproblems to be 
.solved, and brances are tagged with English descriptions of the sub- 
problems they lead to. PATTIE uses these transition phrases to determine 
the path the student is taking through the graph. Other branches (leading 
to LEAF nodca) are tagged with PL/1 statements and represent the final 



EKLC 



-77- 

82 



step in refinement of a particular task, namely translation into the 
target programming language* 

The remaining features added to the basic AMD-OR graph 
formalism are special branch or node types. Special ERROR branches 
allow the problem expei^ who develops the graph to provide specific 

hints to be given to the student only in certain contexts* These 

" 

ERRCTK brai!iam!« Swum either AND or OR nodes, and lead to LEAP 

nodes vhich have error messages attached* ERRCB branches tagged with 
transition phrases ("expected" ERROR branches) correspond to bad 
approaches the problem expert felt students nere likely to attend at 
that point, and the error message can explain why that approach is not 
good* Untagged ("universal") ERROR bradches lead to error messages 
which simply suggest explicit actions which are piobably needed at that 
point* 

A second special brauich type was needed to handle the common 
practice in top-down program development of intermixing l>artial progra mmin g 
language statements with English descriptions of refinements. This branch 
type may be tagged with PL/1 statements and marked so as to be displayed 
as soon as the node is encountered, but not trcLversed. 

The final special feature is the FROG nod^, which allows 
invocation of a subroutine before it is programmed in detail* When : 
e- countered in the refinement of one procedure, the FROG node acts like 
a LEAF, but also causes the interaction control program to stack the PROC 
iS the root of the new procedure's subgraph, for later detailed development 
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6.1.2* 8tttdant'>tutor Interaction 

In relation .o the AND-OR graphs the successive refinement 
process corresponds to tracing a path througn the grajAi froza e root 
to some subset of the LEAF nodes vfalch represent a solution program. 
The exact path taken Is determined by sizggested refinements input by 
the student. At any given time, the student is 'actively refining only 
one task^ the "current task." This current task is represented in the 
refinement graph by a single node, which PATTIE determines the correct- 
ness of student- suggested refinements by matching them against branches 
in the graph at and below the current node. Once the student has 
described all the actions needed to refine the current task, a new 
current task is selected by simply traversing one of the described 
branches from the current node to a new current node. The order in 
which these branches are traversed is detemined by the control program 
using a depth-first traversal algorithm. 

The means by which PATTIE \may interact with the student is 

the display screen of the PLATO IV te^inal. Figure 2 is a copy of 

\ 

the screen as the student sees it. The\^upper 20 lines are the "program 
area" and contain the developing r olutloh program. The lower part of 
the screen is the "scratchpad," the area wl^ere the dialog is conducted. 

On the left-hand side of the program area are a series of 
"task names" indicating the relationship of ea^h teusk to others in the 
solution, exactly as the relationships between sections of this thesis 
are described by the section numbers. Task names^are assigned when the 
refinement task is initially described, based on the task name of the 
current node and the current node type. Each refinement at an AMD node 
receives a task name composed of the task name of the^ AND and a suffix 



OIFTERM: PROCd.Z) ; 
DCL(T,Z,BFOR ) CHfiRi 

1. 1 DO. (NDXZ,LST«? ) FIXEDi 
NDXZ -INDEXCT.Z) i 

1.2 IF NDXZ mB 

THEN RETURN('a') I 
2.1.2 BFOR -SUBSTR(T,NDXZ +2) I 

2.2.2 LSTrtR -INDEXCSUBSTRlT.NDXZ +3) , •«') I 

2.3 separate the exp from the rest of the 



2.3.1 IF LSTPR .0 

2.3.2 THCN exp' is the rest of the striht 

ELSE 

2.4 simplify the parts and return the finswer 

■ END piFTERM; ; 

I now refining task 2.3 j 
What else must be done to refine tYr^ current task? 
> 



HELP nv.) ava»ls)Li<; if wav-ted 



Figure 2: The student's screen display 
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indicating the nuir>^er of the branch matched by the refinement. At OR 
nodes, on the other hand, the task name of the refinement is simply that 
of the OR itself, since only one branch leaving the node is ever traversed. 
These task names indicate the relationship between tarks described in 
the successive refinement process. 

y^ni^in the program area there are three distinct siibareas. 
At the very top are programming ^language statements, corresponding to 
refinements -which had been described well enough to be tran^^lated and 
displayed as code by PATTIE, Immediately below these is the natural 
language description of the current task. Finally, at the veiy bottom 
of the program area are other tasks awaiting further ref Inemer/o, This 
is essentially a stack of refinements described at AND nodes during 
solution development, but vhose corresponding branches have not yet been 
traversed by the control program. These refinements provide : . cont : rt 
in which the student can devise his refinements for the current task, 
but they needn't yet be considered in detail. 

The scratchpad area is where PAll^IE accepts student inputs, 
displays hints, or reveals the anticipated refinements once available 
hints have been exhausted. This interaction goes on (solely in t :2 
scratchpad area) until a correct refinement for the current nbde is 
input by the student. At this point, the refinement task description 
is moved to the program area, the exact location depending on the 
current node type. 

If that current nods is an OR, the student only needs to 
input a single refinement which matches one of the branches leaving 
the node, and the control program's actions are correspondingly simple. 
It simply determines which branch leaving the node is matched by the 
jcudent injnit, replaces the current tajsk ^descritrtion on the screen with 



the siiggested refinement, and traverses the matched branch to a new 
current node. 

AND nodes, on the other, hanC, correspond to points in the 
solution process where refining a task requires describing several 
separate subtasks. As each refinement is accepted, its description 
is moved to the program area stack. When all needed refinements have 
been described, the current node Ts pushed on a stack of active AND 
nodes (i.e., nodes with all necessary refinements described, but at 
least one branch leaving the node untraversed), the top refinement 
description on the program area stack is moved up to become the new 
current task, and the leftmost branch frcxa the node is traversed. This ^ 
corresponds to a depth-first traversed of a part-'.cxaar path through the 
solution graph. 

Of course, an essential part of the tutorial process is 
providing hints to the student when he makes a mistake. There are 
several levels of prompts coded into the dictlog routines which 
provide slight hints to the student. These are dependent on the 
current node type and may be superseded by more explicit hints contained 
in the solution graph, if such are available. Such specific hints may 
be provided by the problem expert who develops the graph by means of 
ERROR branches. If an expected ERROR branch leaves a given node, the 
error message the branch leads to will be displayed only if a student 
input matches the attached transition phrase when that node is the 
current node. ERROR branches are not examined during the lookahead 
matching process, to avoid potentially misleading hints J If the brat&ch 
is a universal ERROR branch, the error message is displayed in response 
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t6 the first wrong input received when that node is the current node, 
and then the prompt sequence described above takes over. 

Finally, the tutor contains a student model based on ^ l .at 
of semantic concepts relative to problem solving and the subject area 
of the particular problem. Each node and branch in the solution graph 
may be tagged with one of these concepts, and for each concept the 
model keeps track of the probability that the student will suggest 
a correct refinement at a point in the. graph ta^jged with that concept. 
This information can be used to provide addition€tl hints to the student, 
or to modify the standard procedure of asking for suggestions and 
immediately display one or more of the desired refinements. 

6.I.3. Natural language capability 

An analysis of protocols between a human tutor and a student 
over the C uine prognimmlng problem the tutor is concerned with indicated 
two things: 

(1) student utterances are short, ungrammatical, av,d relatively 
isolated from each other; 

(2) students use only a smaJ.l number of patterns in theii 
utterances (both typed and verbal) 

Item (1) ruled out the use of a linguistic-based understanding system, 
and item (2) provided hope that the tutor could make do with the simple 
dialog understanding system provided by the PLATO .IV author language, 
TUTOR. ' 

Essentially, this facility is a keyword recognition, pattern 
matching scheme. Aii author specifies a vocabulary, consisting of a 
number of disjoint classes of synonymous wordc (groups of ^'content" 
word?:) and a list of words which are allowed in a student's inputs 
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but vhich carry no meaning ("ignorable" words)* Elements of a 
synonym class may be single vords or "phrases," -which are a series 
of two or more words which must appear contiguoixsly. Phrases provide 
a simple means of handling common idioms, and may consist of ignorable 
words, content worciS appearing elsewhere in the vocabulary, or 
completely" new words. 

tutor's facility attempts to assign a meaning to typed 
inputs by matching the input against a series of stored patterns. Each 
pattern consists of representatives from one or moi*e of the classes of 
synonymous words in the vocabxilary. Since there are usually many ways 
of expressing an idea in natural language, it is .frequently necessary 
to attach mor^r than one keyword pattern to a single "meaning list." 
For example, since keyword order and number of keywords are Important 
in a pattern match, if it was desired to assign the same meaning to the 
inputs "a brown cat;" "a cat that is brown," and "a cat" (assuming "cat" 
and "brown" are content words and other words are ignorable), the meaning 
list must include the patterns "brown cat," "cat brown," and "cat." 

One of the biggest drawbacks of a synonym-class approach such 
as this is that a word can have several different meanings in different 
contexts. Since.no word can be in two classes (except as part of a 
phrase), classes which. contain the same words must be coalesced. This 

---j-g^^rQ^j[^^go4^^ft_<>A,T ^.^ ftn^ount of ambi^ity into the meaning attached to 

some inputs. Fortunately, a node in the refinement graph provides a 
well-defined context which helps reduce this ambiguity caused by merged 
classes. The most likely student inputs at a node are exactly those 
which correspond to transition phrases tagged to branches leaving that 
node, or nodes slightly lower in the graph. Therefore, if an input 



matches one of these branches^ there is a high probability that the 
intended meanings are the same. 

After severed, improvement iterations, this sld^le scheme 
allows the tutor to understand about 80^ of student inputs using a 
vocabulary of about 1500 vords. 

6.2. Sorting lab and verifier 

There are a nxunber of reasons for exposing beginning 
programming students to the concepts of program correctness. 
In particular, the discipline of structuried programming depends heavily 
on correctness proofs of program segments, and personal experience 
indicates inventing loop assertions for a program greatly increases the 
programmer's \mderstwding of his routines. Unfortunately, few beginning 
programmers have the ability to ccurry out a correctness proof of their 
program, whlQh suggests that a program, verifier would be a valuable aid 
in teaching introductory programming. 

Many of the verifiers which have been written, however, require 
intervention by the user to direct the activity of the verifier, which 
is not acceptable for beginning students. So it was decided to develop 
a program verifier which could verily simple programs without inter- 
vention from the student. To accomplish this, the particular domain of 
programs the verifier accepts was limited to programs for inplace sorting 
of, the elements of a one- dimensional array. This domain was chosen for 
two reasons: 

First, sorting programs ^e among the most used ^ampl^es in 
introductory programming courses, and second, every program verifier 
constructed so far has verified several sorting programs, which provides 
a standard for comparing this verifier to previous work. 
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The varification system consists of three major components: an 
editor, an interpreter, and the program verifier. Let's consider each 
of these in a little more detail. 

6.8.1. The editor 

The editor €l11ow& programs to be written in a prograimning 
language with primitives especially designed for sorting (Figure 3)* 
In^plAce sorting routines must conserve the keor^ they are sorting; 
hence the language provides two primitive operations for moving keys 
( exchange and insert ), and does not allow assignment of values to the 
keys of the array. Successor and predecessor functions on the indices 
of the fiurray, as well as a special scan, statement, provide sequential 
access to the elements of the array. The language also includes if, 
-while , and call statements. All procedures are allowed to be recursive. 

The editor is designed to facilitate top-down program development 
The program is internally represented as a tree; deletion or insertion 
of a subtree between any tvo nodes is permitted at any time. Also, the 
editor insists that the student complete each statement before inserting 
another (e.g., the endwfaile of a while statement must be properly inserted 
before going on to other statements). Finally, since the language is 
sufficiently modest that nearly all its statements may be recognized by 
the first character, the editor completes program statements as soon as 
the statement type is recognize!, allowing the programmer to concentrate 
on the program being developed. 

The as^'ertion language provides two predicates concerned with 
arrays, namely sorted (s,t) and array( s,t). Their meanings are sorted 
(s,t) <rs> if s < r < 3 '< t then x(i) < x(j)} array (s,t) < array (u,v) 
- ■ <s> if s < 1 £ t and u £ j £ v then x(i) < x(J). The language also - 
contains predicates (<f/ =, >, <, >) for relating indices- of the array. 



< ptr > <^ < ptr > ( + 1 ) 
exchange < key > vlth < key > 
Insert < key > below < key > 
vfalle < boolexp > do 
endwhlle 

if < boolexp > th^n 
else 
endif 

scan up \ vlth < ptr > from < exp > to exp > 
^dbwn\ ,3 — - 

endscan 
procedure < name > 
call < name > 



4i 



Figure 3: Programming language statements 
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An assertion is then a sentence composed of these basic predicates and 
the connectives and and or. Notice that it is possible to express the 
negation of a pointer predicate In the language, but not an array predicate 
(i.e., sorted or array ) . The student is required to provide an assertion 
statement for each loop in the program: a loop body exit assertion 
(B^^) and a loop exit assertion (LgrjQj)* Examples of such assertions 

are in Figure k* 

6.2.2. The interpreter 

The interpreter is capable of executing any program written 
in the programming language. During execution, the status of the array 
being sorted is dynamically displayed, along with the location of the 



various indexing pointers (Figure 5)* Only the currently active 
procedure is displayed; as each new procedure is entered, that procedure 
is displayed along with the diagram of the array segment and a stack of 
procedure names giving an invocation trace. Both assertion language 
and programming language statements are executed, and the truth or 
falsity of the assertion language statements is indicated. 

6.2. 3» The verifier 

Because this verifier is only concerned with a limited domain, 
it is faster than other program verifiers in existence. There are two 
reasons for this: first, since it is impossible for the program to destroy 
keys, the verifier only needs to prove the keys are sorted; second, because 
of the specific domain, the verifier has been designed to prove theorems 
which occur frequently very quickly, while perhaps taking longer than 
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1 procecMre sort (n) 

« i i N (a XSI i MMn) i XN+1 

2 sscan doiun with i from n to 2 v 

3 scan up with j from 1 to i-1 

4 i f X j > ^ j 1 then 

5 exchange xj with xj+1 

6 else 

7 endi f 

* 1 i cj I <: N & i xu+1 & fkiM c) i s(x+i;n) 

B _ end scar^ 

36: 1 <* X ^ N & r=%(l;i-l) i s(XiN) " 

.9 endscan 

5(i;n) 
10 endproc 



>>> what next? <<< 



Figure k: Sample sorting program 
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9 

8 13 



7 12 



6 11 
5 
4 



3 7 



2 4 



1 1 



1 procedure sort (n) • 

« 1 ^ N & X0 i A(1|T4> i XN^I 

2 scan -<io^»n with i f 

3 scan up with j from I to i-l 

4 if > xj*l then 

5 exchange xj with xj+l 

6 else" . 
►7 endif i . 

8 endscan 

* ^ < I i N & ) i S(liN) 

9 endscan 

* s(i;n) 

10 endproc 



:;: executing 7 » :;; 

array diapla^ ' exe«ut ion cain b« nesLuned 



Figure ^: Sample execution display 



usual to prove less typical theorems. 

The verifier is coo^posed of three distinct subsections. The 
first of these, the verification condition generator, is responsible 
for creating theorems to be proven from the student's assertions. For 
each loop in the program, two theorems are generated as follows A 
loop body entry assertion (Bgjj^pjjy) is generated from the body exit 

assertion (Bgjjjrp) by backward siibstitutipri, assuming C stands^ 

for the loop condition, we muiat prove the two lemmas: 

(2) Bjjjjj^ wad iC igg^es L^j^(the loop exit assertion) 

Beginning with the bottommost and innermost loop, and working outweurd, 
such lemmas are generated for the whole progzw. These are then passed 
to the second section of the verifier, the theorem prover, ^^tolch proves 
or disproves all lemmas for the program. The third section is a counter 
example generator, ^ich will provide the student with counter exiranpies 
for any false lemmas. . " ^ 

Note that the theorem prover is the subsection ^Ich is - 
specialized for sorting programs; the lemma generator is a comp>leteIy 
general routine. Also note that the v^^rtfier will always terminate, and 

■ •■ .. ; ' ^•^-^'^ / '•■ ■ ' 

indicate whether the pro^^^i'S^ correct or incorrect with respect to the 
given assertions ^-^'^ 

"^.2#U. Performance of the system 

The editor and Interpreter alone- provides a htg^ily^r lilstru ctl ve 
sorting laboratory which has been well received by flinidaits who hive 
tested portions of the system. The ver if i^^r alone, for those programs 



vlth Which it has been tested, has proven to be the fastest system 
known. (It must be noted, however, that other verifiers can handle 
relatively arbitrary programs, while ours is limited in its domain) • 
A typical bubble sort routine, for example, requires nine CFU-seconds 
to verify. Iftaf ortunately, this may require as' long as 30 clock-minutes 
during periods of heavy system load, which severely haujpers its usefulness 
for instruction. 

6.3. Program anomalies detectable by an automated system 

Beginning programming studeats learning their first programming 
language normally have a very "narrow" view of "the problem solving process. 
They learn the function of each rf the individtial statements in the 
particular language but are not familiar with all the language features 
and lack the insight to select the most appropriate language constructs 
for a particular problem they must solve. 

Among some of the reasons why this occurs are: 

- Lack of experience, 

- Teaching technique, 

- Lack of desire to expand their own programming ability (usually 
caused by lack of interest), and 

- Misunderstandings or misconceptions. 
Because students: 

- Start coding before they understand all aspects of the problem. 

- Program piecemeal and add "fixes" to patch up inccMplete algorithms 
instead of restructuring or changing the basic algorithm, and 

- View their program as a series of essentially imconnected 
statements without reflecting on more global aspects of their 
program 
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"Poorly stipuctured" programs are often produced.* Here, "poorly 
structured" refers not only to control flow but also to inefficient, 
ineffective, or erroneous data flow. 

An automated system capable of performing the global flow 
analysis that the student fails to do is clearly appropriate. Such a 
system capable of: 

* Detecting program anomalies, 

- Giving detailed information about the anomalies, i.e., 
helping the student understand what is wrong, and 

- Helping direct the student in correcting the anomalies woxad 
be a valuable pedagogical tool. 

6.3.1. Data collection 

A set of four machine problems given as assignments in a 
beginning programming course of approximately 60 students has been 
collected. The course used Fortran as em implementation language and 
was directed toward Engineering students. The final solutions (those 
handed in for grading) are currently being hand analyzed for program 
"defects" dealing with: 

- Programming style, 

- Efficiency, and 

- Language and algorithm misconceptions. 
A report presenting: 

- A categorization of these "defects",* 

- Reasons why students produce such "defects", 

- Statistics on "defect" frequently, and 

- Which "defects" are automatically detectable 
wi2JL be completed within a few months. 
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6. 3«2, Tachniwas 

- The thesis involves the use of global flow analysis ^ 
techniques (both currently existing and newly developed) to detect 
anomalies in programs. 

A flow graph corresponding to the student *s source program 
is produced. This flow graph is then used by iterative tecjhniques 
similar to t^ose developed by RUdal ["A unified approach to global 
program optimization", SIGACT SIPIAN pp. 19^-206, Oct. 1973] to 
perform each of the specific global flow analyses. 

A uniform iterative global flow framework has been developed 
which encompasses most of the "standard" (l^e., '"Live" Variables, 
common stibexpression elimination, doodnance, etc.) and newJ^ developed 
(i.e., unreferenced data, unititialized variable, transfer variable, 
etc.) analyses. Since these techniques do not involve "interval" 
analyses, the underlying program flow graph need not be reducible. 

6.3.3. Specific program anomalies 

This section presents examples of some of the anomalies to 
be detected. Each can be detected by the techniques mentioned in 
section 3 without any knowledge of the user algorithm being implemented. 

Figure 6 is a subroutine inqplementing the binary chop method 
of root finding and will be used to present specific examples of anomalies 
to be detected. This is the type of code many beginning Fortran 
programmers produce as a f ineiL i>roduct (i.e., turned in to be graded). 
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LI SUBROUTINE BIHCHP(XL,XR,EPS,13ELTA,R0OT) 

12 YL = F(XL) 

L3 YR = F(XR) 

IP(YL*YR.GT.Oj GOTO 10 

L5 20 ITER = 0 

I6 IF(ABS(XR-XL).LE.EPS) GOTO 30 

L7 XM = (XL4-XR)/2. 

L8 m = F(XM) 

L9 ITER = ITER + 1 

LIO DELTA = ABS(XR-XL)/2. 

Lll PRINT,ITER,XM,DELTA 

L12 IF(YL*YM.LT.O.) GOTO UO 

LIB XL = XM . 

LlU YL = YM 

L15 GOTO 20 

Ll6 .UO XR = XM 

LIT YR = YM 

Ll8 GOTO 20 

L19 30 ROOT = XM 

L20 10 RETURN 

121 END 



Figure 6: Binary chop routine 
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6.3.3>1# Unreferenced data 

Uhreferenced data occurs vhen: 

- A value, D, is assigned to a variable, V, and 

- That value is not referenced by any statement of the 
program. 

This can happen in a combination of 2 ways: 

1) Variable V is assigned a new value prior to a 
reference, or 

2) The variable V is never referenced, i.e., aix "exit" 
is encoTintered prior to a reference. 

Example: 

At LIT of Figure ^, a specific value is assigned to 'YR\ 
However, there is no ancestor of LI? which references 'YR 

6,3.3.2. Uninitialized variable 

A variable, V, referenced at a spiscific statement, S, may be: 

- Totally uninitialized 

i.e., no execution path from the beginning of the program to 
S assigns a value of V, or 

- Partially ininitialized 

i.e., there is at least one execution path ftom the beginning 
of the program to S which does not assign a value of V. 
EN:an^le of partially uninitialized variable: 

Consider 'XM' referenced at LI9. Assuming 'XL' and 'XR' 
are sufficiently close upon entry to the subroutine, i.e., 
ABS(XR-XL) <=EPS, then the flow of control might be (LI, 12, 
L3,Llf,L5,L6,L19,L20). ThlB execution path leaves 'XM' 
uninitialized when referenced at LI9 and, thus, an erroneous 
root is returned. 
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6. 3- 3- 3* Code motion 

Code motion can be suggested as a correction to certain 
anomalies vhen the student asks for help. For instance^ assume the 
partially uninitialized 'XM' at L19 has been detected. The suggestion 
to move L7 between L5 and l£ can be automatically generated.- 

6.3.3«^- Transfer variable 

A variable Is a transfer variable If: 

- The value of an expression X Is assigned to V> and 

- At each reference (normally only one) to V vhich contains 
the value of X, the defining components of X have the same 
value as when X was assigned to V. 

The reason for detecting such a situation Is that the assignment 
of X to V can be eliminated and the expression X siibstltuted for 
corresponding references to V. Although such a substitution probably 
produces a more efficient program, this is not the major reason for 
bringing this to the student's attention. The primary motivation is to 
help the student understand how data flows through his program* 

Exainples : 

1) 'fCxR)' assigned to 'YR' at LB can be substituted for 'YR' 
at (thus, eliminating L3). 

2) 'XM' assigned to 'ROOT' at L19 "^^n be substituted for 
•ROOT' at LI. This eliminates L19 and since no explicit 
action must be performed before returning, the 'GOTO 30' 
at j£ can be replaced by 'RETURN'. 

JML_ 
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:!here ar^ several situations >M even though detected, 
should not be presented to the student. Two sugh' situations are: 

1) The transfer, variable is assigned^ the ycaue o 
expression requiring computjsttipn; (i.e«., not Just the 
value of another varicible) outside^ a loop but is 
referenced inside the loop* Clearly, thfe value of 
the expression is invariant to the loop Sad its computation 
has been placed outside the loop for execution efficiency, 

2) The transfer variable is assigned the value of an expression 
requiring compixtation and is ref*^'rejdced more than once. 
Thus the computation would have to be ^ performed more than 
once if a substitution vere done. 

Example: 

•F(XM)' assigned to 'YM' at LB can be substituted for 'YM' 
at Iil2, Ll*^ and LI?. However, this produces two functional 
evaluations each time throu^ the loop when only one is . 
needed. 

6. 3. 3* 5* Initialization inside loop 

When building ^ "IP" loop, the beginning student often places 
the initialization of the loop inside the loop. The two concepts of 
code* motion and transfer variable can be used as a partial solution to 
detect and correct this situation. 

Exainple: 

As the subroutine is currently structured, * ITER* at L5 
is a transfer yariabM Ci^e^^^^ '0* assigned to * ITER' 
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at L5 can be substituted for 'ITER* at L9). If L5 Is 
moved out of the loop, say to 'ITER' is no longer 

a transfer variable because the data referenced throuj^ 
•ITHR' a'G L9 now has two sourcest 
Thus, \t movement of the assignment to a transfer variable to 
a position outside the loop changes its statiis, it is a candidate for 
a misplaced initialization* 

6.3»3»6» Common expression detection 

Studen^is often calcxilate expressions with exactly the same 
value several places in their program. Such duplications can be 
automatically detected. The purpose of bringing this to the student's 
attention is not to produce more efficient code (since an optimizing 
compiler will eliminate such redimdeuat computations^ but to help the _ 
student better understand how information flows through his program. 

Example: 

The value of 'ABS(XR-XL)' computed at l6 is exactly 
the same as that computed at LIO. A temporary variable 
can be ussd t>o transfer this value to ithe two places it 
is used. 

6.3.3.7. GO'ing to a 'QOTO' 

The 'GOTO' is svandard tool used (especially in languages like 
Fortran) to handle momentarily unresolved fictions. When these actions are 
finally resolved, the student fails to perform simple optimizations in 
order to slisplify the control structure and produce- a more understandable 
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prograi. A csiass of such "defects" is the explicit transfer of control 
to an unconditional transfer of cortrol. 
Example: 

The 'GOTO 10' at ih can be replace by 'RETURN'. Such a 
form is more easily understood by scmeone reading the 
program and better reflects the intended meaning. 

6.3«3*8. Local variable in a tjarameter list 

Students vill often place a local variable of the' subroutilne 
in the parameter list. This can'bften be automatically detected even if 
the corresponding argument is actually manipulated in the calling routine 
(although computations involving the argument are noraially completely 
absent). 

Example : 

Tbe variable 'DELTA' in the parameter list at LI is probably 
a local variable. Since 'DELTA' is assigned prior to any 
reference, it cannot be an input variable . Assum^Mj^he 
value returned vo the calling routlt.e is never ^^^^^|Bk 
(see section 6.3.3*lO> It cariiot be an output ^^^^Hv^ 
it can be concluded that 'DELTA' is a local variasll^^^ 

6.3.3.9. Modification of inwt TParameter 

It is generally considered a poor programming practice to modify 
an input parameter in a subroutine (of course, a parameter may be used 
for both input and output). Such a pract iceman cause erroneous results- 
(if the corresponding argument is referenced expecting it to have its 
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original value) or excess computatiuus to recalculate the original value 
..f the argument. 

Bxaiople: 

'XL' and 'XR' in the parameter list at LI are clearly ^ 
input parameters since they are referenced before they ^ 
are assigned. If the values returned to the calling routine , 
are referenced, the programmer may incorrectly assume^ he 
Is referencing the original input values. If the returned 
values are never referenced (i*.e., the parameters are not. \ 
output parameters) program emomalles may occur ^en the 
subroutine is used in a different environment. , 
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> lancpttge dMign and •pplleii them to «n Invtitlgatldn of contar^ oonftrueti 
>r InienotlYQ eonpatlng, pftlettlMPly In O m ft Ur Aldad antruotlon (CAI). 
I M axpeiPi— Bbal approach to lan«uw dotikn. a dMlfpar rooogntMa tba 
mn •Umant la progra iw i i ^ atid attaivti to aefaitra an optiaal daalgn 
r an oniplrieal Invaatlgation of languasa oonitmota. Throu|{h oasrafiOJy 
jcumantad, thoroui^, and raplioabla axpariaanti, diai^iara oaa praiant 
>jeetlv« avldanea to iupport olalaa Kbout langnaga faaturaa and atyllitlo 
snaidarsblona. In a formal approach to » daaigner " 

leognlaei tha thaorctleal foundation! of programing languagaa and 
btaoq^a to aehlava an optlaal daalgn by a apaelflcatlon of tha propartlaa 
f language eonatrueti In order to axpoae iraaknaaaaa, Inconalatanolaa, 
nd daalgn flcira. A better underatandlng of the ayntax and aaBantloa 
f language oonatruota can nake It eaaler for a language detlgner to 
bjeetlvely aee vhat featusea are really dealrabla. 

Tbeae approaohea to language dealgn are not only appllo«ble 
n an Invaatlgation of propoaed language oonatruota but alio In an 
nveatlgatlon of language daalgn prlnolplea. Many llita of dealgn 
vlnclplaB exlBt, but notlona auoh aa '^alapllolty" or "ualfomity** 
hat are generally Included In tbeae lUta are laauffleieBtly defined. 
. formal definition of theae prlnolplea would faellltata identification 
nd conBlderntlon of language featurea that violate baalc daalgn 
irlnclplaa. Moreover, experiment a can be applied In aa attaapt to 
fbjeetlvely validate dealgn prlnolplea ao '.hat language duaigaara 
laa confidently apply them. 109 
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7.2. Awwpoacih * 

* ■ f % 

As a means to explore experlaeotel and f oxnal Is&jsuage 

desiflp, 0. W. Bdbley has desiffMd a new prognamlng langnage. called 
BOL CM)l«yi 197?a]. KAIL was originally motivated by a desire to ' 
Improve TUTOR [Shenrood, 19J^]f the author language for the FLATO IV 
CAI syatas [Alpert and Bitser, 1970]. In RAH, a selector construct 
[Bribley and Hansen, 1976] Is Introduced to handle CAI answer judging; ^ 
this construct also unifies selection and iteration and si^bsumes most 
typical high level language constructs (e.g. , if -thett* -else, i<hlie« 
repeati - imtU ). Figure 1 gives the essential syntax and sesuntics of 
the KlilL selector. A static exception processing sohssM is also 
Introduced to handle frame sequencing. 

These KAIL control constructs were tested in two experiments 
conducted on-line In a CAI environment, and the results Indicate that 
they are likely to be psychologically sound. In an experiment on the 
KAIL selector C&ribley, 1975b], subjects attempted to understand and 
answer questions about two short programs. For one group of subjects, 
these programs wers written in 8, a language containing the selector) 
and for the other group, they were written in A, an ALQOL-llke language. 
In the other experiment on frame sequencing [lUblay, 1976a], subjects 
debugged and modified a svibstantlal CAI lesson sbout 500 lines in length. 
One version, T, was written in a TUTOR-llke language, and the other 
version, K, was written in a KAIL-like language. 

These KAIL control constructs were also examined through a formal 
definition of their semantics, and their properties were dearly exposed. 
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^ selector Li^tatene^t-sequen^^^ control dliolces] 

[statement-sequence ix)op»control choices] 
control If expression 
loop-control -» ^lle expression 
"♦ until expression 
choices I relation /expression : statement-sequence 

-♦ I relation expression : statement-sequence choices 



semantics : 

The selector Is executed in 5 steps 

1. ' Execute "fihe Initial statement-sequence* 

2. Evaluate the coxrtrol-expression and save its value in a 
temporary, t. * 

3. Test each choice in turn until a "selected choice" is found 
such that 

(t relation choice-expression) 
yields tmie * 

k. Execute the statement sequence in the selected choice. 

3* Execution now continues as follows: For If, exit. For 
while ^ if no selected choice is found, exl^ othtrwlse, 
return to step 1. For until > if no selected cboiee is 
found, return to step 1; otherwise, exit. 



Ill 

Figure 1: Essentials of the KAIL selector 

t 
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An axioEAtlc approach was applied to the KAIL selector^ and the KAIL 
exception processing scheme was defined in terms of a behavioral 
model. The TUTOR exception processing scheme %ras also formally 
defined and coopered with the KAIL scheme. 

7.3. Conclnaions 
7.3-1* Design principles 

As a result of the investigation of experiment^ and. fbt^ 
approaches to language design^ three hasic design principles jByoly^ 

1. Itoiformity, : r 

2. Separability, and < 

3. Locality. 

These three principles are proposed eis a possible basi^ for an infozmal 
approfush to language design. 

The uniformity principle stxggests that languages ou^t to 
be designed with a one-to-one relationship betwe^ j^^ 
In a program written in a uniform language^ a single semaixtic notion 
consistently has the sam^e syntactic form. Mbreover, a sugle rulQ 
applies to each language construct independent of its context* In a 
nonuniform language^ there are several ways to fucpress a singly 
notion or various possible meanings for a syntactic fom depending on the 
execution history^ so a programmer has more to consider. STonunifbrmity 
leads to more decisions and thus more probabilil^ of error. 

The separability principle suggests that special purpose 

composite structures msy be harmful. , The advantage of a composite 

. ' .* 

construct lies in its pcwer to produce a desired effect with a minimal' 
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amount of code. Most high level language structures are cc^oi^oslte 
constructs; and so long as no prograxmner needs an unavailable component 
of a con^lilte construct^ all Is well. Th the unfortunate situation ^ere 
a needed component Is unavailable^ the language designer may be willing 
to extend his language. If not^ a prograxmner wiiuld have to obtain the 
component indirectly by "prograaning uround" the problem^ if this is 
possible; otherwise he would have to use a different language. To 
supply syntactic forms for many composite structures would cause language 
constructs to proliferate. To force the programmer to indirectly separate 
composite structures^ on the other hand^ would result in code that is 
more difficxat to understand and maintain. Separability suggests that 
a few general language constructs are better than many constructs that 
have only specific utility and that caution should be exercised in the 
creation >f special purpose constructs. 

The locality principle siiggests that language features should 
be as permanent and local as possible. Locality aids prograxmers because 
it structures information. When a programmer has a large amount of 
information to consider^ any mechanism that structures this information 
or restricts it so only a small subset needs consideration is most 
helpful. When a programming language encourages locality^ it reduces 
the amount of text a programmer xmist consider in order to determine the 
effect of a language construct. Furthermore, it imposes a structxxre 
on information accessing methods and restricts the variability of language 
features whenever possible. Locality also facilitates modularity and is 
particularly valuable when a program must be modified. 

In [EUbley, 1976b] these principles are formally defined. Moreover, 
the experiments conducted in this research lend support to these principles, 
particularly the sequencing experiment. 




7.3«2. The experlmentcLl approach to language design 

The restilts of the selector experiment support the hypothesis 
that progranmers understand the KAIL selector more easily than an 
equivalent set of traditional constructs. S language subjects answered 
more questions correctly than A language subjects and also thougjit they 
initially better understood the S-favored program. Statistics on the 
number of questions initially answered correctly, average time taken to 
obtain a correct answer, an^ initial and final self -evaluations are all 
in the direction of the S language. No performance stajbistics favor 
the A language. 

In the sequencing experiment, the results generally lend 
support to all three basic design principles. In T, the behavior of 
a procedure is context dependent, but in K, the behavior is indejjwihdent 
of context. T subjects introduced errors due to this context dependency 
when they Improperly inserted new procedures. This lends support to 
the uniformity principle. Several observations support the separability 
principle. T subjects had difficulty separating composite constructs, 
and in one instance, none of them were able to find a way to "program 
around" a particular problem. The experiment also supports the locality 

principle because T subjects consistently failed rmmmij gj-pbal statua 

information when they attempted to fix one of the bugs. In K, this 
information was local and caused no particular problem. 

The experimental approach to language design can produce 
scientific evidence to support claims about language features and design 
issues as shown in both experiments. Throu^th empirical tests, designers 
can gain assurance that their language features are psychologically sound, 
and they can gain confidence in language design princip;Les« 

114 
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7.3-3- Ebcperlnents In the EEATO environment 

Since these experiments were conducted on PIATO, they also 
Illustrate the applicability of an on-line methodology for conducting 
experiments in programming. Several advantages can be gained by 
conducting experiments on-line in an interactive, Ckl environment. 
There can also be several disadvantages. 

The advantc^es include: 

1. A controlled teaching environment, 

2. The ability to interact meaningfully with sub^Jects 
during an experiment, 

3. Individualized sequencing, 

k. The ability to gather highly precise data, 

5. The ability to impose strict timing constraints, 

6. Assistance in grading, 

7. On-line editing capabilities, and 

8. On-line execution capabilities. 
The disadvantages include: 

1. Cost, 

2. System failures, and 

3. Subject oinfamiliarity with the system. 

In general, the experiments profitably took advantage of the 
PIATO environment. Time and space .'Limitations, however, prevented full 
exploitation of potential advantages. 
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7.3*^* The formal approach to language design 

An application of an axiomatic formalism to the KAIL selector 
helped clarify and concisely specify several general observations. The 
KAIL if and case are semantlcally identical^ and all is coni|^ex compared 
to the other control types. The formalism also revealed similarities 
and differences among if > ^diile j and untile and led .to an investigation 
of another possible control type. 

The semantics of both KAIL and TUTOR exception handling/were 
defined in terms of a special purpose abstract machine. This behavioral 
definition shows that the KAIL constructs adhere to the locality principle 
better than the. TUTOR constructs. Moreover, the fornwOism shttws how the 
TUTOR constructs violate both the uniformity and separability principles. 

The formal approach to language design can objectively reveal 
properties of language features as shown in the formal definition of the 
KAIL control constructs. It can also objectively expose weaknesses ^and 
inconsistencies and provide insight into why some language featuJres are 
better than others. 

7.I1.. Summary 

The results indicate that further research in experimental 
and formal language design is likely to be fruitful. These methods can 
be applied to obtain objective evidence to support claims about language 
features and design i^f^neB in general. lb would be particularly valuable 
to further apply these methods to obtain additional support for the 
three basic design principles. These could then be confidently used 
as a partial basis for reasoning about and designing programming 
language features in general. 
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8. Use of ACSES In Instruction (r, g. MbntaneUi> Jr., E. R. Steinberg) 

An Introductory computer science course at the Iftilverslty 
of Illinois ordinarily consists of 2 large lectures taught by a 
professor and a smaller discussion taught by a teaching assistant 
(TA) each week. The Ijectures introdugej^oAt new mateHal^ >Aiile 
the^^tUsGus^ions. are small classes in vhich TAs answer questions and 
help students with thedrncrGopuier^p^ Since someivfaat more 

than.pne-half of the lecture time was typically spent on FORTRAN, 
it was initially decldied to develop a sequence of PLATO lessons to 

teach FORTRAN and to use these lessons to replace one^^lecture a week, 

1 

throughout thel semester. 

Work on the lessons was begun in the fall of 1973 hy 
students in an honors course. During the course of the projeot, 
most lessons were written by students. In addition to being in 
computer science, many of the students had some teaching (or teaching 
related) experience as teaching assistants, consultants, and graders. 
All the lessons went through numerous stages of testing followed by 
revisions, corrections, and improvements. Ultimately lessons were 
polished by highly experienced staff members or students.*'' The 
lessons were not designed according to some particular theory of 
instruction because it is not cloar that a suitable one exists 
(Anaatasio, 197^)* It was felt that the best way to arrive at a good 
set of lessons would be to encourage varying styles and techniques, 
and to ultimately choose the best lessons on the basis of their 
effectiveness or on student preferences. Of course this could 



^e authors would especially like to thank Professor H. 0. Friedman, 
Jr. , Sandra Leach, and Jeffrey Barber for '<;heir invaluable help in 
this area. 11?^ i 
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(and did) result ^ s^nie lessons vhich had to be completely rewritten, 
bxrb it avoided the possible trap of using a theory which migsht not 
apply. More detail^ about the development of the lessons are given in 
Montanelli (1975), while Barber (1975) reports an indepth study of the 
design^ evaluation^ and subsequent revision on the basis of data 
lected during use^ of one lesson. 

8.1. Fall> 197^ 

The initial use of 12 of the lessons to replace clasrr^ocwi 
instruction occurred in the fall of 197^, although optional, voluntary 
use had occurred for some lessons durit\g the previous sp:*lng and 
suraner. A relatively small class (50 students) taught by the first 
author was selected for the first actual test. In order to obtain 
some comparisons between the lessons and ordinary classroom instmiction, 
the class was randomly divided in half, with one-half receiving the 
traditional two lectures and the other having a PIATO lesson replace 
a lecture each week. There were three interesting results from this 
early experiment. First, questionnaires administered at varying points 
during the semester indicated that although students seemed reasonably 
satisfied with PIATO early in the semester, the comments made at the 
semester's end Indicated some dissatisfaction. Four possible explanations 
were: 1) early in the semester students found PLATO new and interesting, 
but the novelty wore off by the semester's end, 2) students were more 
concerned with grades by the end of the semester, 3) earlier lessons 
were In better shape than later ones, and k) a computer memory (ECS) 
shortage made it impossible for a group of students in a room of 
PLATO terminals to use many different lessons simultaneously. 
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tTndoiibtabljr each of these explanations played ° some part In student 
attitudes^ and of course^ little could be done to ,clu^ 
of the first tvo. However, it was the case that some of the earlier 
lessons had been better tested than isome of the later ones. Thus some 
additional inqprovemenbs for these lessons were indicated. Also, the 
shortage of ECS could have bad a Icurger effect at the end of the 
semester, because as students fell behind or needed to review, they 
created a demand for many lessons at the same tlrMy imd taat wasn't 
possible. » 

The most encouraging result was a correlation of . ^8 between 
the amount of time spent in the required nAZO lessons and the ^sourse 
grade. If this was a cause and effect relationship, it illustrated 
the usefulness of the lessons. - ' ^ ^ ^ 

The final interesting result was the lack of a significant 
difference between the two groups on achievement variables. With 
some of the problems encountered, this result' was good, in spite of 
the fact that all observed differences favored the non-FIATO groups 
vith two exams almoat shovring significant differences •* the .05 
level. Other results showed similar, low drop rates in hoth groups, 
and no relevant differences from the previous year's class on the 
results of a student rating of instruction form. Mbre detailed 
results appear in Montanelli (19T5). 

Three possible methods for improving the instruction gotten 
through using the FIATO FORTRAN lessons were identified. They were: 
1) Increase ECS; 2) revise lessons, especially to include more 
exercises and other interaction; and 3) consider more strongly 



encouraging students to use the FIATO mater l8J.s. On line data Indicated 
that on the average^ students probably did less than one-half of the 
assigned material. A computer-managed instruction (CMI) program 
(Anderson et al, 197**) bad aJLready shown that PIATO could be used 
successfully to increase studlh't perfozmance through CMI only. 
A later experiment in fall 1975 > considered this question. Also, 
ECS was added in January of 19T5> solving one problem, and revisions 

of lessons were beg\in along lines indicated. 

ft 

The study concluded that CAI materials initially written 
by students could replace some lecturer cn the FORTRAN language 
in M ir.+vo.^^'''^tory computer science cour^**. Hov/ever, it was obvious 
that more effort mtust b(2 s:f>ent on lesson development and evaluation 
tbari WC3 originally suspected. 

8.2. Springj 1975 

The purposes of this evaluation were fourfold: (1) to get 
baseline information on students' a:btltudes and to determine if there 
were changes during the semester; (2) to assess the data collection 
in CS records; (3) to provide guidelines for revision and Improvement 
of seme of the lessons; and (k) to provide recommendations for improved 
implementation and integration of FIATO into CS courses. 

8.2.1. Students' attitudes 

It is particvilarly important to evaluate studant attitudes 
when a new technology is Introduced. A positive attitude is a necessary 
though not always sufficient condition foi* learning. The student's 
attitude must be sufficiently positive that she/he is willing to try 
the CAI lessons. Student attitudes can also serve as a valuable 
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resource of information for revising and improving both the lessons 
themselves and methods emd procedures in course Implementation. Although 
PIATO had been widely used in a variety of couriaes at the University of 
Illinois, it was miticipated tnat most of these beginning CS students 
vould"^ not have had prior experience with it. What was their initial 
reaction to the prospect of using PIATO and on vhat basis vas this 
made? As expected most of the students^ in the saxnple (71 oizt of 99) 
had not had previous PIATO experience. Responses to an open ended 
question revealed that about 39f> of the initial reactions were positive, 
kki) indicated fear or displeasure, and 2k^ could not be interpreted 
positive or negative. 

Thus, although most of the students had not tiad previous 
experience on PLATO, their expectations were not negative. For the 
most part, they were uncertain or favorably disposed. Their comments 
revealed some concern and some confusion, not knowing what to expect. 
Their sources of information about PIATO were mainly other students who 
had courses on PIATC and instructors in this caarse. This information 
may not have been entirely relevant because other courses may have 
used PIATO in different ways than CS 105, e.g., supplementary drills 
or cooqputer management of instxniction. lb seems, therefore, that 
students need some specific information about CS 103 and PIATO. 

Three questions were asked at both the beginning and end of 
the semester to assess possible changes in attitude toward PIATO per se. 
Students were asked to rate each statement on a 5-point scale, from 
strongly agree to strongly disagree. There was no statistically 
significant difference between scores conqparing the entire initial 

p 

Questionnaires were handed out randbnOy to students in different 
PIATO sections. . 122 
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pit of 99 to t)M Md of •6Mttor auvlo of 75* Vor iiM.>lwr« 
lgnlfto«nt itff«me« ifhtn tlw dKto Uaitod to tbt 96 studmtn 
flUod out qaMtloiuMdrot both in Juuuy wd April. BQfii^r, 
Ota bo leoB ttcm Tabl* 1, tbf yxoportioM of li^^ 
ortMa la Junniy di0X«M0d by AprU. 3^ *1^ ^ ^^ 
t iuoo tt "m^i^ til^ it If 4«b»MBislB8^ 
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tiM mt«ri«l'*' w thb/'vott' lnort«at;'ft««piiti«« i^^^^^ 
ir- of tlw ■«todtaat« 1^ 
»> -*IAJto U. of -I... eoluppo -if ^«t:j^:ito^ 

I llSi( tb: airoid it. ' IM iMn. r^ : ' . / 

leontly dtfferont Ikoii tlM^^ aBawBawft 

» «tad«ita la 23 eoar8U . (Bio8el» ISh^)^: Al^^ 

Lt that tbo PIAIO l^seBtctioa vas ao«t offeetlv* forvtb^-f^^ 

rafred, 30]( fttlt the vrosontation voold Baim be«i.%^^ 

reetive by lecture aad/or textbook. A f eir gtndintt' -I'iwwltf ijfT - jaiat. 

9^ favored RAIO becense the leeturea >Hre 'Voii^eaavf -CX^^ 

fds^ ltSiasn't that KAXO was so great/ but that bom ctf^^^-^^ 

re so poor. 

muit "bugs" students nost about FLAIO is 'Mlien thay eooe a 
ang" distance and the systea is 4o«a. tkif ortttnatft]y» dnie^ 
days ~^^df the saaMster the systSM lias doim asst of the ti^ 

With respect to the lessons* alud aMli aire apst inrttated 
inadequate response judging. Thai l4is|Kin m freq^aantly aa 
bheraoaw ima fortcoap t resjponse tlae vas too slcnr aad it ;Ma 



Attltudtts of CS 105 Student! to FZASO ftt 
Beginning and Bad of Spring, 197? SeoMter 



glnnlck 



AOBEB GR 
S^JEKfflOLT AGBEB 

JMr.» AHl.** 



.(A 
.16 
.1»8 



.or 

.17 
.65 



DlSAaElE^ OR 



JAR.* 

.73 



.01 



.58 ; i^^^^ 



.10 



1 Coaqput«r-lMuied education is nothing but an exiien^ 

2 Cdpputer^lMiaed edncation deibumanlsea the ,8tud<^» , 

3 I ajogred (or expect to enjoy) using PIATO^tldf^ l^^^ 



JAW,* AHl.»» 



.27 .12. 



.gS . .,; ..13; 



124 
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'ecmfuslng. Sdme coaipla.ined that leMOM yere too'ilong/^^^^ 

POTTIUVH artlimfltic Mid PORERAii fbrnfttjfclag.^ Me^tt bveraU rating for the 

. - "gbbd'^;' The 6^^^a^t!Biefia dLt(^ t6x 6oix^ 

' '■ \learn. '-;. liie ■sbidents-"\)hordid:''th^ m^dltm'-;.;};^!-^^^^^ 
".'■r; -.of" liiatrfiK?blQtt----ove'r 'ikiWoer^hUB^^^ the\;>;,:'v-;,V''';' 



-'if- 



able to iiork at their bwh pcMtie cuaa; tl^ 



HATO ±p a lefeture^: mateiriai , ^Zfii^tfr^^ 12^t 

said'they ^had no;. preference, , ^ -..-^j-^ ? 



8;2«2« ifcta coitecilon 

^ i " The data cpliection in CS, records kept i;rapk 
' of tlines: a stadent 1^ islgned^^^^^ to a lesson and 
on a.iesson. Data did not teU 

• lesson and r^ it or had done part of the lesson Teach time he - 
signed in* Data, reyealed that^ the average time spent on some ;of the 
lessons in class -tends to exceed the allotted class time. Some exanples 

are 'given;: here,-. ' 

lesson mean time SrD> ^ . ' rahge . timeig. in :iesson 

. fortif 1^.63 17*»f 18-82 > Si 37 

lodps 6l.»i6 19.^ 30-116 Wu 

^^^V; - -6b,79 jrt-^^ SS-l^^^ 

f ' •' •' ; * ': . •*• ■.*'•.,.•,, ,. • • . •; t .'1..,.;.." . .•• • - • U, 

t^- -//^--- '.-r'. •■■ ■ ^ .V -119- ^ * 



Another useful aspect the data collection vaat the^ 
of last date the jstudents had signed on. A qulci gliu^ 

ctor to check up on attendance. In April a random saaqple of 26 
ts in each of the six C3 105 sections ms -qhosen. Qcii^ SOfft ot 
these 1^ students had signed onto the sy^ 
2 veeks. The reasons were not McertMned ih^^^^t^ 

8«2«3« Lessonst instructional design and leia^^ v ^ 

It seemed th*A :so^ 
the interactive fealrore of ^^^i^^ Leissons: Varied; 1^^^ 
exercises provided^ reg^oirement^o^ spebific criVerlbn^^^^^ 
in order to propeed^ and frequency of sefte^bf^:«^ 

A study was designed using lesson 1 
the effects of learper control^ plMemeiit^ or 
versus varian copy handouts of the lessbnv Uhfprlii^ 
system was down most of the time during widch^^^^t^^ 
have taken place. It was also down during ; the prcNsei^^^ 
the teaching assistant was to have jset up ioqplementatlon^^^^^ 
unahle to check out her work. ^ : ; -^^^ *.v 

One should not assume results* of the 10-minute q^izze^^^^^^ 
dependable for drawing conclusions. However^ they mi^t provide some 
tentative insights. t . >v 

Mean Scores on a 10-point Qjoiz Given.by 0^ in Qutlz Sections 





Coercive 


Non~ 
Coercive 


Handouts 


Frequent sets 
of exercises 


T-76 


• 7*26 


6.1Q1 


Few sets of 

exercises 


7*06 


6M 


7.51 



• It is difficuXt to understand wlQr students with fewer exercises 
on handouts did better than those who had more exercises. (The reverse 
was true on PIATO. ) Note that the hifijiest ©cores were obtained by 
those who had more frequent sets of exercises and who were in coercive 
condltioni3. It was decided to repeat the experiment again the next 
semester, when it was anticipated that the system would be more stable 
and the lesson thoroughly tested. Results are summarized in section 

8.3. . ' 

A second experiment was set \ip for lesson f ortftatl . The purpose 
was to see which instructional conditions are most facilitative. The 
factors in the 2x2x2 design were (l) coerciveness (required or 
optional), (2) instructions to do problems or do them correctly, 
and (3) size of exercise set (2 or h of each type). Do students follow 
suggestions in the instructions about how much practice they should get? 
Table 2 shows that students in the optional conditions did more I-and F- 
format problems then in the required conditions. (This may have been 
due to an artifact of the lesson. Required students were not given 
the ox^portunity to do more than required. ) But in the E-foxmats 
optional students did fewer problems. The same pattern emerges when 
students are told how many problems to do correctly (Table 3)- In the 
I- and F- formats optional did more than suggested, in E- format they 
did fewer. This may have been related to problem difficulty. Table k 
shows that all students did a rather high percentage of problems 
correctly in I- and F- format exercises. However, in the . E- format 
exercises, the percentage of problems done correctly was only 51^ 
in reqtiired conditions and Uk^ in optional. It is apparently the case 
that when the problems are not too d^lcult, students follow suggestions 
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Table 2 



Ifean Ntoober of Problems Bone by Type of 
Exercise and Experimental Condition 



I-fomat (N=3l6) 

Reqtilred 
. Optional 

P-format (N=300) 
Required 
Optional 

B-format (Na2l(2) 
Required 
Optional 



Small Sets Given 
Do Do Rie^t 



k,2 
8.2 



6.5 
9.7 



6.0 
8.1 



5.4 
6.k 



7.0 



11.1 



Large Sets Oiven 
Do Do Ricctxt 



8.2 
12.k 



12.0 
13.5 



12.0 
il.8 



9.1 
12.2 



13.8 



l8.t 
12.8 



Tible 3 



Condition 

Required 
Optional 



Number of Problems Done Correctly 
Instructions 





Do 


Do Right 


I-fozmat 






Req:uired 


5.2 


6.1 


Optional 


8.2 


7.9 


F-format 






• Required 


7.8 


8.9 


Optional 


9.1 


9.1 


E-format 






Required 


k.O 


8.6 


Optional 


5.0 


5.5 



Table k 

Percent of Problems Done Correctly 



84.2 
76.0 



FORMAT 
F 

86.1 
77.6 

128 
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E 

50.8 
43.6 



for. Ijow BBwh to do, or practice even more. l*i<in soM^^^ 
is encountered, they do significantly less than rn«i^«d. 3^^ 
be pointed Cfut that optional students did a lower percent correct and 
this was statistically significant* Hq^ 
significant. That is, a student perfoimiilis; at 76^6 <^ 

do axqr worse on a svbsequeiit achievement : 
at an B^i accurate level. There may he cb«msider^^^ between 
studwits operating a* 51^t and Wti leyelA. Ih^ 
levels would seem to he satisfactory to 

Apparently a reasonable minlimm reqairement for pafa^ice 
is not Abrasive to students. Althotijih hot ess^ial for less difficult 
concepts, it seems necessary for more difficult ones. Al80> the 
difficult problems (E-format here) should be aMCcon^wuiied by some form 
of corrective information in feedback and/or help secpiaices. 

8,2.U. Classrocm observations (course Implementation) 

Students took notes on lessons. A questiUaaM distributed 
after students had con5>leted lesson fortlf . Results showed that about 
2/3 of the students took notes on lessons fortlf and fortarith and that 
3/U of the students who took notes knew that the material was covered 
in the tesct. 

Whenever the classroom was visited, a proctor was seated at 
a terminal near the door. She/he wore no identification nor was there 
any sign on the terminal. Students had no way of knowing that a "human 
being" was available for help. Furthermore, proctors generally were 
busy prograaping or playing a game on PLATO, so that if a student did raise 
a hand for help, it was unlikely to be acknowledged. 



BeccmnentotlonB 

1. More extensive and better comminlcatlon should be eatabilsb^ between 
instructors and students as veil as between proctors and students. CS 

^ 105 Instructors mig^t help create a more positive attitude by orienting 
students as to the goals of the KATO lessons in CM3 105: how niuch time 
they will take/ what they can expect fraa the lessonsj iriib will help 
them if there are problems^ and so on. It Is also Inqportant that 
they tell the students that , ITATO is used d^ 
courses; therefore^ previous negative expesrlm^ 
applicable here. 

Provide a large sign for the; proctor to put^^o^^^^^ 
terminal so that students knoir that a person is in the rQ<M 60id 
available. Proctors should be oriented to the responsibil^ 
walking around the classroom periodically. Also^ *Uiey slumld tM>n^ 
worked through each of the lessons themselves. 

The attendance^ or lack of It^ art FIATO sessions should be 
investigated, it may be part of an overall student syndrome of 
generally poor class attendance at this time in the semester, it may 
be in part a reflection of student attitude toward the usefulness of 
the FIATO lessons at this point in the course. 

2. Revise data collection to include the time it takes to cciq^lete 
each lesson^ nuinber of times the student has completed entire lesson^ 
and number j^^v times the student has entered lesson. 

3. Lesson rexi^lons l.d be aimed at minimising student fjriustration 
and using mor^ . a Interactive capability. The length of time 
it takes students to coiaplete a lesson must^be compatible with ion^^ 

. of time allotted* Some of the lessons apparently take inore time tha 
anticipated because, students take notes. For iong lessbns^ a of 

ERIC 



be. feasible!: ,;aai<w^2.sesi?ions4^^ :^".;n--''>^->; 



bt 'tiPoVia^^ ojP the text :pMt8 of ^lessonis; and: tti^^^ 

re^ tU^^^^ co^iiig to:F^^ and to spiimd 

^' A- "declsiozi'- l3dM ib ■ be ''miid^ 'a]3.dut'!theu'^SM^^ ' igpial of ; each 
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ierstands the concept; Hcwever> liroin the point of .: view. ^^^^^ n^ye!; > \v: : 



learner^ a different seq^ a i^tr^ 

^."'fv' ■ ^> ;-^vv; •^^^''v>/^■■v■ 

Consi^rtfble b^ia^it^^^k^ 

the i^dandie of : a f irsi-rate lectiarer' ^o haas iinAi^t^^ to ; 

auch iqrp^ An iMjK>rtant atep in leBapn/:r9V^.8iott p^^^^^ 

the c^^ to obaeHTve a atudent (or €^ peraon^:^^ 

coUeagdecr) do. the leaaon* Thlfs ahould provide :c6naid««flxlj6|^^ 



about the kind of reapohisea to exj^ and the 1^ of d^ 

ajbudent i^^ Leaaona ahould^:^^^ " 

reapcwiae judgingi aliowlrig for at broad apan of correct alieitiativeai 

'^^^^ - V;^ 

exerciaea avidlcible* \ A good paradisn miglit be. to reguto a minlxBum , ; 

number of xnh^^ done correctly, idth t^ option ta^to 

/as .the.atudtot ''chTO8ea> " ■ '■ ^ - ; 

U. student attiti^ were igenerally favorable to CS IQ^ .on^^ a!here/ ; : 

waa appcffe^ no performance decriaaidf^^ compared to preyioua ; a^rieate^^^^ Z\: 
Tlie l^it ted^^^^^ revisiona aa ap tov reach w 



8.3. Pan, 1975 

After a year's experience using the FQRXRAN lessons^ and 
vlth seme revisions planned for the sxnamer of 1975 > it vas decided 
to conduct a large scale^ controlled experiment in CS 105^ In the 
fall of 1975, in order to determine the effectiveness of the lessons. 
In order to control for effects of instructor and time of day, four 
CS 105 lecture sections were scheduled, two at 9:00 am and two at 
10:00 am. Students at each hour were randomly assigned to one of the 
two sections, and one section at each hour was arbitrarily chosen to , 
use FIATO to repOLace one lecture per week, while the other had two 
lectures. Finally, two professors (A and B), neither of whom had ever 
used FIATO prior to the start of classes, were wslepaed to the 
sections so that professor A taught a FIATO section at 9:00 ud a hcm<» 
PLATO section at 10:00, while professor B did the reverse. (A fifth 
section, taught by a third professor, used HATO, but was not Involved 
in the experiment.) More detcdls concerning this experiment are available 
in Mbntanelli (1976). 

The three hypotheses of this study were: 

1. FIATO students would enjoy the course more, and give it a stronger 
recommendation to their friends. 

2. PLATO and non-PIATO students would perform equally well on exams 
and homeworks in the course. 

3. The drop rates in the two types of sections would be similar. 

In answer to the question (from the questionnaire administered 
with the final exam) : 'If a friend were taking CS 105 next spring and 
FIATO and non-FIATO sections were offet^d, what would you recommend he 
take?' PLATO students strongly recommended PEATO (112 circled /definitely 



PIATO% 88 ^FtATO if convenient Us hiad /no reco?^^ 

'lecture if conypnient % and 21 said • Mef Initeiy ^^^^ On the 

other band, non-FIATO students nmtr^ (their 

were 29, SS, 91, 26, 19), or even^ showed a, 

: order to xidnpaM ^^axi^^ 
analysis of variance . ms: ccniiuied ( 

cm coBxputer progri^^^ Tdtldff isrw ftiun^y-^fl^^ 
wre nearay-ldaitieSli^f^^ 




only U (i^y vj^^ 
and a8^lU^)^^^ f^ 
';"'^':y;;-:'"3tudiBnts 
their friends 



Even If ^ the ^'ertra' ^25rdrbpB \iXi]iiike^'^IA£p'^e^^ 
they wuld have a snali^ effect on thei^?ai^ , 7 : r f 

recdnniending PLATO/; and cmly 36 of them arecoBto^^ ; a ^ 

sh<iuia he remaiked that the 

distance to CERL* (iSafortunately the terminals m 3ocatedi:on the north 




what they thoujjit: vere the worst feature of 



edge of campus in CERL, about a mile f rem most ccpnerce cb^ 
37 checked 'Lack of hui^ and 31 checked •EIJa!6^^i^^ v 

the next two most frequently dtiecked responses. Thus the nus^^ ; 
was unfortunately cut of bur con^ L^^^'l^^^^^^^^^;^^^^;^^^^^^^^-^^^;;; 

The second hypothesis was not rej ected, due to ^ nearly identical 
scores on exams and machine problems. TheTC; is no reasb to^^^susiiect that 
the HATO drops were poor students. Hbwever^ if the dropped PIATO 



students irsre beloir average, they could not have ^^h^ 
effect on the results to alter the pbvlbus conoluslon. TbdLs result 
is: certalAiy in agreement with most studies of the effects of CAI« 
In fact; lAen Ja^ and Wells (197^) sujhreyed the effectiveness 

of alternative instructional n»dia, the^^^^^^ 

• • . • "the equal-eff ectiv^ concaoision !s€m^ tb hei broadly liorapect 
for moist alifeerhate fl^^ instrtuHlii^v^ 

and^ suggested studsf^^ 

a mador advantage d* (yi^ thai once weii> 

a tea*bbttk or Aji a.res^ 

iwtoich studients liked tlie least were t v 
a q]ai2 system has been begun; When compietQ^f ri^-'^^ . . 

to each student at the cooqaetion of each lesao^^^^ 
written by the authors of the lessonsy and Iri^i^ 
(Uscburaged from look^^ 

written from the same objectives that were use^^^^ wlte the lesi^^ 

Bie restilting g:uiz scores will not ^on^jr t^^ 

tmderstand l^e material ^ich the lesson is supiBk) 

tell Instructors and lesson awthors how w^Li^ -^ 

Thus^ continual Is^ovement is possible, axid p^^ 

materials will be as good as the best lecturer,^^ 

than maxty. . _ ; : -'^ ■-^■/■^^'^^ ,/.-v r\ 

On the other hand^ the hypothesis etboiit eq^^ drop taites 
wets re;)ected. This was a sxirprlslng result, espebicpJ^ when't^^ smaller 
experiment a year earlier (under worse condltioAs^) showed no 
EEt>mver^v the^^^^^ lupre beexi a speciskl 

relatively 8mall> elective course ^ w^^ 



psychblbgy end similar fields. These students were more involved and 
tnb^eWted In the experiment, and they may have stayed for that reason. 
On the oth« hand> CS 105 is a required course for freshmen in the college 
of cDimerce^ and the students were jfcesunabiy less interested' in long 
term educational goals (for theMelves as well as for the PIATO materials) 
Howeyer^ although this drop rate was. disturbing, theiw were a. few, likely ; 
reatsons for it, all pf which could be fixed. For one tht»«,^^^1^^ 
three weeks were confusing for the sttidente because they had pre^ 
in a course which they ejcpected would consist of two lectures; and a 
discussion each week/ ISwitead, three-f^ of them hid ift^l^cture v 
cancelled and had to sign up for a PIATO sscbioh inrstead. Ui^e sections 
caused a lot of trouble, as scane were schsdtiled for week-endd, and many 
students' complained that they were unable ic meet any of the rema.inl^ 
available PIATO times. Althou^ most of thlfi confusion was necessary 
due to the nature of the experiment, in the future students will 
pruregister for PIATO sections Just as .^^ : any other class.- A second 
possible cause for the different drop rates was that for the first 
few weeks, PLftTO students were required to do the -pr^gi^^ 
In one of the online. Interactive conqoilers. Althouigji It wis thought 
that thl's would be fun for the students, the compiler gaye very. poor 
response time because of the amount of processing going on to check for 
errors after each student keypress. Finally,; ^(^^ 

in part to student dissatisfaction with the two pocwr^^^l^^^ were 
later rewritten. Students had nob been systemiBAlcally 

the lessons before, and the; relatively negative reaction to two of U - 

Vthem' ' was q^lte.. surprising. : , ; ' "■'l^'''- • ' ■,•■^^^..^;_■ 

Another possible explanation for the highw^^^^^^ 

HATQ, is that some students ,(W lbiJ^& GAI . : . 

ERJC" <i^^^M^ic&iii^i^^^'.r^%izP:^ 



yiil alimys have this problem. The authors do hot fetX ilUtt the > 
large differences found here could be attributed to HxLb reason. Bovever, 
some data on this question Is reported below^ for CS 10? In spring 197<^« 

jn suivnazy, this experiment shciwiaA thisbt XUITO less<MUi can be t ^ 
us^ to replace one lecture a week ija an i^^^Hn^ ooi^puter progrjB^^ 
course/ Students learned as much and -pret^fKBiniO 
sections. The remaliri.1^^ problems are;; : l) ja ^ drop . 

rate on HifLTO? and 2) Can inj^tructlon be i^ contlJlued ' 

development of the C!AI materials? A V 

At the same time as the large experlU&ent-^^^i^ 



In CS 105, some smaller studies were being cohch^ 

class. In the first of these. Barber's (iSfTS)^^^!^ 

arithmetic was used to test studeiit attitude and pezT 

after using different versions of the leason. ; Ea^h:^^^p^ 

was randomly assigned to 1 of 6 experlme^ cprqt^^ the^^^ / 

lesson in one of tluree modes (PIATO coercive, ^^ i^^ 

haridotrts) and under one of two cotiditlons of 

less frequent). PIATO toearcive stiidents were f^ 

in the lesson, while the 'PLATO zum-coerclve p 

ANB t^'^llain the answers to the online exercises^ withcrut^^ 

them, jliudout students were locked of 

which essentially consisted- of copies of the displays the ot^iw^si^ - 

saw at the texinincd. The 

exercises i^aced in vaxd^us parbs^o^ 

ended with a few exercises. The "less^: 3^ 

was simply the original lesson ^Aiich bojitadned a;t^ 

drills in various places. ^ w >^ ■ ■■.■■'r'^ : 
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• ji: oe.'-'iio. -iitfiific^iifc'; difsrirttii*!;- '■r«t^.:iS^^ 

Mili ih^^^^^ handottt itudenbs- or betiirtirii 
iftlonlfig iproisps^ do eitbav the 

rerttl dflgm after the leeioii or on a queatlon tm TOBBUUI arlthmetie 
an exam given three iredca later. 

There were some Intweetlng differences In attl'tades found 
a qiaaitlonnalre distributed lamediately fbUfllrtng th« nA!CO leBBlon. 
d^vagr analyses of variance, node (FIATO vs» nvitten) hy questioning 
ore vs. Idss freq^ient), were run on the questionnaire items. There 
re significant (P(l, 77) - l£.8, p < .001) differences hetneen EIATO 
1 aon-nATO groups in Aether the students vould reoonmend SCATO or 
bandotrt to a friend. PLATO students recoomended flATO, liliile non- 
ASO students irere neutral. Other statistically significant results 
nded to shotr that FIATO students vlth less frequent questioning vere 
ss happy about some parts of the lesson than the FIATO students vith 
re frequent questioning or those vith handouts. 
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the 

conDiented on each answer^ but the handout student had to look up answers 
and comments in the back. The HATO version ifas perceived to be fiun^ 
terbertalnlng;^ and more interesting by 13 ot hi students iiho responded 
and was credited vith making it easier to learn by 11 sti^ 
apparently had dlfi'erent escpectations of CAI than a textbook. Th^ 
complained that 'V^u can^t ask it questicms. ^ The same ims true of the 
handout J but nobody expected it from a text* . 

More firequent questions on HATO gave the students^more 
confidence that they had learned the material ud a stronger feeing 
that the feedback helped their understanding. There ma no evidence 
of such differences for the handout students in the two questioning 
conditions. 

Student behavior was essentially the same in the HATO coercive 
and non-coercive conditions. Contrary to expectation (Baxberj 1975; 
Az^derson & Faust ^ 1913) 9 students under learner control engaged in 
appropriate learning strategies. The coercive stiadents did not balk 
at the requirements that were imposed. The attitudes evidenced on the 
questionnaire were consistent with this behavior. Students did not have 
a strong feeling that they pref ered to make their own decisions sbout 
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1)61^1^^ the quiz mav; bave 1>eefi d^^ 



A i^elatd^^ 8UC& sij^iisi^ 



the method of priesentatlon. More cosq;0.ex content inie^t h^ve resulted 
in perf omwince differences. | 
The second experiment in GS 103 involved the ^^l^ 
students vould do more vork in the lessons and thus achieve better 
understanding of the course material if they were req:ttire|d to do the 



lessons* Tn order to test this hypothesis^ thenstttdentjs ; in CS^l^^ 
were randomly divided into three groups* In Oxxmp 1, do|.ng the PIATO 



lessons vas not counted as part of the students* grades .j In the other 
two groups^ doing the lessons counted 5i> (Group 2) and 1^^ (Group 3) pf the 
grade with other factors down weighted accordingly. Thef expectatiion 
was that increasing the degree to which the lessons couxjted in a grade 
would increase studying the lessons> and thus increase l^aarning. 

Based on 80 students (26 or 27 per group) who jremained in 
the course for at least 3 weeks and took the first exami the nuniber of 
FIAXO lessons completed (using time in lesson as a conpiietion criterion) 
is f presented in the first row of Table Although theafe 10 about 1 chance 
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Table 5 



Means on Performance Data of Three Qroups 
vlth Varying Percentages of Their Qrades 
Determined f rem Conqpleting PLATO Lessons 





Group 1 

0?t 


Group 2 


Group 3 


Probability^ 






2 

Means 






Ihanber of PLATO lessons 
completed 


7.7 


8.5 


9.7 


.095 


Rtainber of PIATO lessons 
coinpleted ignoring drops 8.3(20) 


9'3{2h) 


10.5(23) 


■.09** 


Total machine prdblism 
points 


136. 


IkO, 


15k. 


' '.50 ■ 


Hour exam 1 vritten 


65.7 


. 6k.2 


70.9 


.29 


Hour exam 1 PIATO 


6J*.7(23) 


60.6(25) 


72.8(65) 


.006 


Hbur exam 2 


50.8(15) 


50.0(23) 


52.6(23) 


.8U 


Final exam 


128.3(18) 


UU.6(23) 


129.3(23) 


.17 



1 Probability of tbe obsarved P value from a 1-vay analysis of variance 
betveen the 3 groups. 

2 Group sizes were 26, 27, and 27, unless specifically indicated, (i.e. l6 
students in Group 1 took the second exam). 
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'"'el-yiw^'no-^to between'.g««T)8-;i todf-a, . of'-group'l did'.'^isjbter' 



.i:;:;^!. jttin-jBr«up:5,-_J^ only 1 statiaticaOly ^sijgaif icaiifc resiat ^todipa^ that ^ 
, ';$iuid<mts ,in group 3 did laetWi^ on l^ii'p^ fSxH hoyxif 

- ^vas glWii^on^PIATO. ^ ^ "'T: 

Se^ral dthier kinds of data were alflo conecrted W 
In the fall of 1975 • Table 6 presents: ? some;: 
the reqjuired PIAIO lepspns, Coluinn 1 gives .the date on 1^ 
: nas taken. ]b general^ lessons were reqid'ed to be ccn^ 

end of the week following that during 'Which /the asslfi^ei^.i^ 
but all previously assigned lessons were required *to/be done before 
an exam, which accouxvbs for the pile-up around Septe^ Coluinn 
2 gives names (somewhat mneumonic) of the regjoire.cw lessons .^^^^ 
reports the number of occurrences of a type of error wtii^^ oecuarred 
when a lesson did not properly return to the operatltig syst<^ 
execution. Lesson fortchar (FORTRAN charactir handltog> liad ia relatively 
large number of such "errors" due to an experimental q^z Vh^Ujb «M 
appended to it (see section on the quiz systan). CcAum If gives 
the nuBiber of tines each lesson <was invoked hy a student in the class. 
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t^^[ CS 103 PaU 1975 - Lesson 

V ; Btamber Btraiber of 

^:v: Bad :-^f Jtwdei^ 





Lesson 


eactts 


uses 


eniiered 




eslnrtro 


2 


129 


' Bk 




fortlntTO 


1 


13^ 


60 


1-^/8 


fortarlth 




197 


89 


i ; 9/15 


forfclf 




2lU 




|: 9/^5 


fortfinfcl 


T 


272 


82 


V 9/29 


foxtarrayl 


5 


177 


81 


|9/^ 


ftebsin 


0 


208 


66 


10/20 


fortarray2 


3 


87, 


U8 


10/27 


blnseareh 


1 


91 


52 


.11/3 


fortfiiib2 


2 


1^9 


57 


11/3 


nunbers 


5 


160 


I46 


U/li6 


fortsubl 


0 




1*9 


12/2 


fortsubex 


1 


95 


55 




fortchar 


19 


122 


57 



Data AeeuBulatad Online 



Rinbar of 

atiiidwta 

coBip3.6fted 


Average 
tSiaa 
per sti^AeBfe 


Averajce -•■.'■vii'^i^^ 
: ' ttine per^ stadMxjl^^l 


81 


" :3i,:.^;' : 




1*9 




:'30-;.^-.,:.:ji^:?^ 


1*8 




; • ■ ■86/^- '--^^^i 


70 


1*2 


■ ■ •» . ■ ■ ^ ■ ^' ■' 


7»* ^ 




:^,73:v.^'-r:-i 


60 


. . 73 ' 








■ ' 56 . 


22 


1*7 


66 ^ 


39 


27 


33 


53 


6U 


68 


37 


63 


75 


26 




70 


18 


30 


53 




32 


l<6 



the average tine for studiDts iiho eoiqCteted the lesson. Thvis^ the 
last coltDBQ Ih T8ible.^6 gives upper boundis (hecaose sone^tutents'^^i^ 
revleirlng) for the avex^e tioes to eoniii^ete the lessons. Note that 
the relatively poor percentage of sttidents finishing fdrtarlth is an 
artifact due to the experiment described earlier in one-third 
of the students signed-on to the system, but vere directed to a. 'haiOmt 
and xirevented from finishing the lesson. Although the generally decreasing 
numbers in columns 5 and 6 would seem to indicate decreasing use of and 
interest In FZAXO. they are also due to a relatively hig}i drop rate 
\ihlch left only 67 students finishing thei course after an initial 
enrollment of 87. This result was largely due to 13 students Vho dropped 
after taking the first exam (in the 6th week), but before the 8th vedc 
deadline. One-half of this exam was given on FEATO, and there were 
several problems resulting in lost exams and frustratiMl students which 
could have caused some of the exbra dropsy Another indication of 
dissatisfaction with the FIATO e:<.am oame from student responses to an 
end of semester questionnaire. Althou^^ only a few students criticized 
FIATO, four said it wij,s fine ept for the exam. Also near the end of 



Althoufi^ no experiments vere run, eezt^n 4ata 
in C8 1Q5 in order to observe routine use of HA!EO. For 
'presents ^^,*e results for 6 semestws to CS Only those 

professors iiho iaugbt at least two seotlonQ, wlth^^^ leut coe o^^^^ 
am Included. Altbough results vltblh the ffeli 197? ssm^^ 
S and V) would tend to >e interpreted^^^^^^^u 
Professors in PLATO sections as ccl^pa^red 

In non^FIAiEO fleetlons, T^sults from -Ctie jUirvi^ " 
Who taugbt PLATO and non-ni^ iBectloiis^ 

support tbls idea. One possible explana- ilon is^lMt botb)^^^^ E 

and 7 reaittrked after fall isrf3, that the;r trUd to j^ve^ 

lecture to both FIATO and non-FCATO grou|)8. Thegr both feiif lihai' their 

lectures to the FLATO sections lacked co^lnulty iiilid ofttm ove^ 

or left gaps with the HATO lessons. 



6.5* Summary 

i ■ . ■ . . 

A few general conclusions can be drawn from the large amount 
of lUta analyzed here. First, FLATO leiETsons originally written by 
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AVpenAlx; Computer Sciance Iabboqb 



a» Saguenelng and Entiy Lessons 



Descrliyblon 
■vluqr Sttj^o the 
Q)BifWiliistibhal Hs^ciBt 



lllkl^er^^^I^^ to the Con^uter T 

IjQrt«roductlon to Mini- 
Langiiage -Sequence 

Intro, to Language Independent 

Prt!^praMn4ng iBeqjience^^^^^^^^^v^^ 

Ibtroductloft to the FL/I LesscHX ^ 
Sequence 

Introduction to the P(Xa!RAH^^ - 
Language ahd Lessons 

Introduction to the BASIC 
Lesson Sequence 

Litroduction to the CX)BOL 
Lesson Sequiance 

Introduction to the: AFL Sequence 

Itartroduction to the LO90 
Lesson Sequence 

Litroductibn to the Data 
Structures Sequence 

lioitroduction to the ^ 
AnalyBls Se 

J^troduotion to the Logical 
Design Se^^ 

Router Lesson for Coigputer 
Scicmce^Courses 



Statue 

:0p6m^i 

operational 
;fufi|t started 
nearly : coiqiiete 



; mrk' Ijtipr^^ 
Just stWPted 
operational / 

operational 

juist/il^^ 

operatiboual . 
Just sti^ 
operational 
operational 
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Name ^ 

'fiPSalm 

epic 

turlng 

cQittpchess 
mazeaearch 

platoiv 



3** 



Ddscrlgtlon 
Ijrtroduptl^ttr r^j^ and r 

CoiBputer Progranffiljag ;^ 
Ifuiber Ife Conputera v^rk progreas 



Stattis V 

^rk in pirogreaa 



Siilhaatlbn of 1^ 2000 ; 
Caloulal^or: .i ^ 

Maze Ti^yerai^ 
, , Miuiuisd. for ffritflx- 

Plata Hardware ud Soffc^ 



operational 

nearly coiB|ilete 
vork J»^: ; 
nearl^: ocxtplejbeV 

nearly 'coiB|£Lete^ 
itei^ in pirogreaa 
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c. Mlnl-Lanyiagas 



NamiB 

Introprog 



pal 



somaga 

pl2d 

csmlnl 

cstrees 

roboint 

robocar 

robostack 

roboback 



Description / 

Introducticm to Mini- 
Language Sequence 

PlctorlcQ. Rrogranpilng Language 
for Children 

ItiSiBifg TAngfi^ - -^ . 

Recursion 

Mini Progranonlng System 
BwDtotype 

Tree and List Manip^tation 
Mini-Language 

Introduction to the Robot Car 
Sequence v 

Robot Car Mini-Language 

Robot Car Stack Aligorlthm 

Robot Car Backtrack Algorithm 



Status 



ijusi started v 



nearly cGoplete 



bpS^at^ 
bpe: 



,,^,itlonal 



opexktibnal 
iroi^l ^ progress 
-ijjork ^in i>3P^ress 

vork, |l& -progress 
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i 



d. Language Independent Brograimdng 



Introprogc 

beglriblock 

detial) 

mes 

recurse 

trigraf 

formlang 
wgrammar 



Description 

totroduetiott to Language 
Independent Frograiming Sequence 

Ilpw Cfhaxtitft 

Begin Blocks 
Decision Tat)les 
Piie SfbceM 
Reciirslon 

Directed Development of a 
Program 

Formal Conqputer languages 
Two Level Grammars 



Status 

ne&rly complete 

nearly complete 
operiybional 
operatllonal 
operational 



oi>erational 
work in progress 

nearly cGmplBte 
work In progress 
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e, PL/1 Language 



Name 


Description 


Status 




In'broduc'blon to the PL /I 
Lesson Sequence 


operational 


pUarith 


PL/1 Arithmetic Operations 


operational 




Strlnff Oneratlons in PL/1 


(q^erfrt^lonal. 


pllif 


PL /I IP Statements and DO 
Groups 


operational 


pnao 


PL/1 DO Statments 


operational 


pllctrray 


PL^ Arrays 


nearly complete 


pllarrayx 


Advanced Examples of PL/1 
Arrays 


nearly complete 


pllproc 


PL/1 Procedures and Subprograms 


nearly complete 


pUio 


PL/1 LIST Input /Output 


ofieratlonal 


plledit 


PL/1 EDIT Diput /Output 


nearly complete 


plleditdrl 


PL/1 EDIT Input/Output Drill 


nearly coniilete 


pllpic 


PL/1 PICTURE Specification 


vork in progress 


pllreciirse 


PL/1 Recursive Procedures 


nearly coinplete 


pllstrl 


Data Structures in PL/l 


operational 
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fovtlntro / Introduction ip tiie fdRTRAH 'wrk in progreiBS 

Lauaguage and Loksotts 

fortavlth Ikitroduction to VCBTBAN, " operationai 

^ ' ' fortarrayl . , One'l)i»M^ ^' -^r'^*^^ 

,;.^orfcarray2 ■ ■ ■ i. ,,Twi::j>im&^^ • 'v ■<ip«pi^^ VB' 

•fortsuibl ' / ■P(»TRAir''sp^^ 

• StajteaBMats' 

. feitftet2 ' 'Advanced- PQEOfRAH -ME5MAT : . '^:;:.v,:',o(pMa*i<^ ■ 

Statement' .■■■.-.'^■" '/ ' •' ' ^ f^' ' ;v 

fiBbsim FORTEUIN POaaMMf Si«^ 

fortctaar CSxaracter Handling in POTOTl^^ 
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g> BASIC Language 



Name 

basieintro 

baslebaale 

basicref 

baslerefl 

basleloop 

basiearray 



Description 

Introduction to the BASIC 
Lesson Sequence 

Introductory SASIC 

Beginning BASIC 

Advanced BASIC 

FOR-NEXT Loops In BASIC 

Arrays in BASIC 



Status 

just started 

just started 
iNork in progress 
vork in progress 
Just started 
vork In progress 
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h» COBdL Language 



coboHntro 

coboUden 

coboledlt 

cciboldata 
cobolproc 
cobolref 



Deacrlpblon 

Introduction to the QOBOL 
Lesson Sequence 

COBOL Identification and 

En!rtxsi»fm*.. 

Advanced COBOL PICTURE 
Clauaea 

COBOL Data Division 
COBOL Procedure Division 
COBOL Language Reference 



Status 
operational 

operational 

operational 

operational 
operational 
work in progress 
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1. APL Langixage 



Name Descrlixbloa Status 

aplintro Introduction to the APL operational 

Sequence 

aplscalar APL. Scalars operational 

aplvector APL Vectors * operational 



1 
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pdpSalm 



Hbchiiie iMCfQ^ j 
FDl6/t SiiBuiator 



"^-^ V-'«{; 

opfmt^tlonal 
i^rk in pz^ 
iiorlc' In progr69S 



160 



- A. 



ERIC 



-153- 



k» Other Langungee 



Name 

snobol 

logointro 

logotest 
logoproc 
logocom 



Descrliytlbn 
SNOBOLl^ 

LISP List Processing 
Language 

Intarbduct Ion to the 

LOQO Lesson Sequence 

LOGO Test Ihstructlons 

LOGO Procedures 

LOGO Cooomands 



Status 

revision needed 
irork In progress 

. just started- 

JIust started 
just started 
vork in progress 
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sortlntTO 

porting 

sortlab 



binsearchl 
lutrostrct 

8tr2 , 

str3. : 

lister 

noder 

treetrav 

catrees 



1^ Information Procesalng 

Deisbrlirblon 
Introduction to Sorting 
Sorting 
3o||i 

3iWLi5^ 5 

•'and:- 



,Strubcti»!BS/:Se^^ 
vlnfonnat 

BbcjpwiMbe* '^j^^ 
■ Ex^periencei^ 

'Traversal; Jof' Binary:;;^ ; -'^i 

Tree^Wd List Mtolpii^ Mini- 
Language • 



Status 

nbrk in progress 
revision needed 
mrk in progress 

• 'woa*^^n-i^ • 
nearly cdnj^ 

bi^ratlbnid. 



operia^ 
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m. Humerlcal Analysis 



Name 

infcronuni 

matmult 

nuiniq.uad 

iiiieqi 

IlneqS 

rootlab 

least sq 

Ilnprog 

montecarlo 

splines 



Description 

Introduction to the Ilumerlcal 
Analysis Sequence 

Matrix Multiplication ^ 

Numerical Integration 

Linecup Equations 

Linear Equations 

Non-Linear Equations 

Least Squares 

Linear Programming 

Monte Carlo Methods 

Spline Approximation 



Status 

juist started 

Sioirk in progress 

revialon needed 
^wrtf in piroi^ 

revision needed 

reyision needed 

operational 

operatl0)a^ 

•opeiWfioi^ 

vork in progress 
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'Simula^ 

racetrack 

payroll^ - 
csslides 



n> Applications 

Description 
discrete Slimilatlon 
Traffic Sinrolation 
SimaXation Games 

PayrcdLl-ftH>grMi - : - 
Conq3uter Uses in Busines^s 



vorlc In progress 
operational . 
operational 

-- ■ operertriwial------™ ~ 

-work in progress / 
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o. System Propraianlng 



Description 

Sicperience vlth Sd^kstra Sema- 
phores 

Illustration of the Oeadlbclc 

Eitpii Stq^ihrLsor' 



Fiiiite State 

Liebcieai^^i^^ 

Top-IXnm Sii^ 

Bottom-Up Analysis of Escpressions 
Code Generation by ^aqilates 



Status 
operational 

operatiqxial 

operatioK^ 

operational 

operational 
nearly CKSiiqplete 
operational 



'J 
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cdbintro 

csbilata 

cflXcoop . 
online 



p# Congniting 8^ 

Introduc^^ 
Compdtintg Be 



CalCoiqp Fla'^e^ 
Remote Terminals 



lipxlc in iax>^^ 

worlc In progress 

qpwatlonai 
vork In progress 
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CL« Logical Design 



Name 


Descrlxrblon ^ 


Status 


Introloglc 

4 


InbrodactloQ to the Logical 
Design Seqiaence 


olteratlonal 


loglcarlth 


Introduction to Digital 
Arithmetic 


operational 


loglcgate 


Coooblnatlonal Building 
Blocks 


operational 


loglcmln 


Minimization of Boolean 
Expressions 


operational 


loglcff 


Basic Seguentlfld Building 
Blocks—Flip ' Flops 


nearly complete 


loglcseq 


Sequential Circuit Design 


vork in progress 


loglccdr 


Ccmblnatorial Problems 


operational 


loglcmsl 


MSI Logical Building Blocks 


operational 


loglchdvr 


Semiconductor Fabrication Methods 


operational 


loglcflow 


Data Flov Diagrams 


operational 


loglclab 


Logic Laboratory 


just started 


boolex 


Boolean Expressions 


vork in progress 


loglceomb 


Conblnations of Logic Circuits 


Just started 
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r# ^CbttPllers 



Name 


BMcrlsrtion 


Btatiis 


reAnatiual 


Befer^ce Muval 

for tha Cta-Line Ccoq^ilers 


operational . 


cursedit 




opei^tlonal 


Wltfl 


JOKSBM . 

Compilen ; 


opera 


pUcoflsp 




under revision 


pllcompe 


Hi/l Ccmpilisr 
vith Idiie Editor 


under revision 


fortcomp 


70R!ERAN Ccmgiler 


under revision 




FORTRAN Cosqpiler with 
Idne Editor 


under revision 


baslccomp 


BASIC Compiler 


under revision 


cobolcomp 


COBOL Coispller 


under revision 


paacedcomp 


PASCAL Coinpller 


under revision 


snobolcoijip 


SNOBOLif and SFITBOL 
Compiler 


tinder revision 


llspcomp 


LISP Compiler 


under revision 
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s# Connmmlcatlon 



Name Description Statiia 

• ■' « 

csccnments Ccmnents between Cl^ Students operational^ 

and Authors / 

cstallc Qn^-Line Consultation with ah operational 

_ Instructor _ . _ 

csmsg Bulletin Board for Course - operational 

Messages * 

csnotes CS Author - Author Coimminlcatlon operational 



169 



Descrlptlog Statttft ,. 

Suggestions on Plato Lesson operational 

Writing Stylp ^ 

Useful 'Material and Cpdi^ . . operational 

Convantlons-for-XS-jtothors - 

Graphical Lesson Structure Design operational 

Mini Programming System operational 
Prototype 

Library of Useful Routines, C!har- operational 
Sets, Micros, Sbc* 

Coding Suggestions for CS Lessons operational 

KAIL Lesson Programming Language work in progress 
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